2024-05-06T00:13:00.580 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-06T00:13:00.581 DEBUG:teuthology.run:Teuthology command: teuthology --description rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nbd} --verbose --name dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi --owner scheduled_dis@teuthology --archive /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201 -- /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/orig.config.yaml 2024-05-06T00:13:00.677 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T00:13:00.759 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201 branch: wip-dis-testing description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nbd} email: idryomov@gmail.com first_in_suite: false job_id: '7692201' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: b45ed4400464709677363de54673bcce970bcb27 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: extra_packages: - rbd-nbd flavor: default sha1: b45ed4400464709677363de54673bcce970bcb27 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-qa sha1: ed0ddec0117930fb8b53c34226cd9519ed394ff9 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 3217 sha1: b45ed4400464709677363de54673bcce970bcb27 sleep_before_teardown: 0 subset: 68/99 suite: rbd suite_branch: wip-dis-testing-qa suite_path: /home/teuthworker/src/github.com_idryomov_ceph_ed0ddec0117930fb8b53c34226cd9519ed394ff9/qa suite_relpath: qa suite_repo: https://github.com/idryomov/ceph suite_sha1: ed0ddec0117930fb8b53c34226cd9519ed394ff9 targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGgXUPNXOyznlxyLiumTfBns3jLkf8YE2UWAXwZi48NTiSQfPm/+LpLX7l8EX+Ypwyzcgs+sBU/B5RG+bdVINbA= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGeSaYtH1UYfLaevIGqmlmRzPXsS8Y2ayZ/xl5tqUhrAj9I0B3Oj06skZ8osS2ZvFR05O1WaNEqiuUcpJsK46QM= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBV9KSFrlRHKzQoOzk+IqVYTf65hy4y5nncjEqhT7OIvDL2saCX7afDO2Eqj33fCjLHn2jedzbnz04DJNLR8xVI= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 holebdy: 512 nbd: true ops: 6000 punch_holes: true readbdy: 512 truncbdy: 512 writebdy: 512 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-05_18:40:04 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-06T00:13:00.759 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_idryomov_ceph_ed0ddec0117930fb8b53c34226cd9519ed394ff9/qa; will attempt to use it 2024-05-06T00:13:00.760 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_idryomov_ceph_ed0ddec0117930fb8b53c34226cd9519ed394ff9/qa/tasks 2024-05-06T00:13:00.760 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-06T00:13:00.761 INFO:teuthology.task.internal:Checking packages... 2024-05-06T00:13:00.787 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b45ed4400464709677363de54673bcce970bcb27' 2024-05-06T00:13:00.787 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-06T00:13:00.787 INFO:teuthology.packaging:ref: None 2024-05-06T00:13:00.787 INFO:teuthology.packaging:tag: None 2024-05-06T00:13:00.787 INFO:teuthology.packaging:branch: wip-dis-testing 2024-05-06T00:13:00.787 INFO:teuthology.packaging:sha1: b45ed4400464709677363de54673bcce970bcb27 2024-05-06T00:13:00.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing 2024-05-06T00:13:01.006 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:13:01.008 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-06T00:13:01.049 INFO:teuthology.task.internal:no buildpackages task found 2024-05-06T00:13:01.049 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-06T00:13:01.066 INFO:teuthology.task.internal:Saving configuration 2024-05-06T00:13:01.083 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-06T00:13:01.098 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T00:13:01.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 00:07:03.551827', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJI2Dkr63pCLgvmX5jJjMZu6t1NIKRH3I94f5rSvfenX'} 2024-05-06T00:13:01.142 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 00:07:03.550481', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB9DMxaO8H0+9lNe+J/01fK//DZA/sF7dCiwcSjTwgmC'} 2024-05-06T00:13:01.160 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 00:07:03.548948', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINLKhjFTXTYgZb079/wAaGa2w0Y+27CQk8RHzn0zCzrU'} 2024-05-06T00:13:01.160 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-06T00:13:01.242 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-05-06T00:13:01.242 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-05-06T00:13:01.243 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['client.0'] 2024-05-06T00:13:01.243 INFO:teuthology.run_tasks:Running task console_log... 2024-05-06T00:13:01.377 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f97295ebe50>, signals=[15]) 2024-05-06T00:13:01.378 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-06T00:13:01.400 INFO:teuthology.task.internal:Opening connections... 2024-05-06T00:13:01.400 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:13:01.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:13:01.482 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:13:01.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:13:01.565 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-05-06T00:13:01.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:13:01.643 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-06T00:13:01.678 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-05-06T00:13:01.685 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-05-06T00:13:01.685 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-05-06T00:13:01.731 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-06T00:13:01.731 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2024-05-06T00:13:01.731 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="22.04" 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=jammy 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-06T00:13:01.732 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=jammy 2024-05-06T00:13:01.733 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-05-06T00:13:01.760 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-05-06T00:13:01.766 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-05-06T00:13:01.766 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-05-06T00:13:01.812 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-06T00:13:01.812 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2024-05-06T00:13:01.812 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="22.04" 2024-05-06T00:13:01.812 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-06T00:13:01.813 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=jammy 2024-05-06T00:13:01.813 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2024-05-06T00:13:01.813 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2024-05-06T00:13:01.813 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-06T00:13:01.813 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-06T00:13:01.813 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-06T00:13:01.813 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-06T00:13:01.813 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=jammy 2024-05-06T00:13:01.813 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-05-06T00:13:01.844 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-05-06T00:13:01.849 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-05-06T00:13:01.850 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-05-06T00:13:01.898 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-06T00:13:01.898 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Ubuntu" 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="22.04" 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_CODENAME=jammy 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:ID=ubuntu 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE=debian 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-06T00:13:01.899 INFO:teuthology.orchestra.run.smithi186.stdout:UBUNTU_CODENAME=jammy 2024-05-06T00:13:01.899 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-05-06T00:13:01.927 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-06T00:13:01.982 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-06T00:13:01.990 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-06T00:13:01.991 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-05-06T00:13:01.993 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-05-06T00:13:01.995 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-05-06T00:13:02.000 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-06T00:13:02.008 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-06T00:13:02.008 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-05-06T00:13:02.040 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-05-06T00:13:02.042 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-05-06T00:13:02.130 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-06T00:13:02.221 INFO:teuthology.run_tasks:Running task kernel... 2024-05-06T00:13:02.261 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-05-06T00:13:02.261 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-05-06T00:13:02.262 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T00:13:02.262 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T00:13:02.263 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T00:13:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:13:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:13:02.269 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-05-06T00:13:02.269 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-05-06T00:13:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:13:02.270 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-05-06T00:13:02.315 INFO:teuthology.orchestra.run.smithi078.stdout:5.15.0-56-generic 2024-05-06T00:13:02.315 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.15.0-56-generic 2024-05-06T00:13:02.316 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2024-05-06T00:13:02.317 INFO:teuthology.orchestra.run.smithi146.stdout:5.15.0-56-generic 2024-05-06T00:13:02.317 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.15.0-56-generic 2024-05-06T00:13:02.317 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2024-05-06T00:13:02.318 INFO:teuthology.orchestra.run.smithi186.stdout:5.15.0-56-generic 2024-05-06T00:13:02.319 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.15.0-56-generic 2024-05-06T00:13:02.319 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2024-05-06T00:13:02.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-05-06T00:13:02.443 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2024-05-06T00:13:02.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-05-06T00:13:02.583 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-06T00:13:02.590 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:13:02.590 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-06T00:13:02.590 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-06T00:13:02.637 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-06T00:13:02.677 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-06T00:13:02.696 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:13:02.745 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:13:02.778 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-06T00:13:02.795 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [453 kB] 2024-05-06T00:13:02.828 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-06T00:13:02.837 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [453 kB] 2024-05-06T00:13:02.885 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-05-06T00:13:02.897 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-05-06T00:13:02.907 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [620 kB] 2024-05-06T00:13:02.914 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-05-06T00:13:02.933 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-05-06T00:13:02.939 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-06T00:13:02.940 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-06T00:13:02.941 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-05-06T00:13:02.963 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-05-06T00:13:02.968 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-05-06T00:13:02.971 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-06T00:13:02.972 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-06T00:13:02.973 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-05-06T00:13:02.973 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-06T00:13:02.973 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-05-06T00:13:02.982 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-05-06T00:13:02.987 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-05-06T00:13:02.989 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-06T00:13:03.002 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-05-06T00:13:03.003 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-06T00:13:03.004 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-06T00:13:03.006 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-06T00:13:03.006 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-05-06T00:13:03.006 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,612 kB] 2024-05-06T00:13:03.020 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-05-06T00:13:03.022 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-06T00:13:03.031 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-05-06T00:13:03.035 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-06T00:13:03.053 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-06T00:13:03.053 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-06T00:13:03.053 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-06T00:13:03.054 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-06T00:13:03.054 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-06T00:13:03.059 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [304 kB] 2024-05-06T00:13:03.064 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-06T00:13:03.065 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-06T00:13:03.066 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-05-06T00:13:03.069 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [453 kB] 2024-05-06T00:13:03.080 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-05-06T00:13:03.088 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-06T00:13:03.088 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-05-06T00:13:03.093 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-05-06T00:13:03.097 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-06T00:13:03.098 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,072 kB] 2024-05-06T00:13:03.111 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [701 kB] 2024-05-06T00:13:03.120 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-05-06T00:13:03.123 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-06T00:13:03.126 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-06T00:13:03.127 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-06T00:13:03.127 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-06T00:13:03.132 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-06T00:13:03.134 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-06T00:13:03.134 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-05-06T00:13:03.138 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-06T00:13:03.138 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-05-06T00:13:03.145 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-06T00:13:03.145 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-06T00:13:03.146 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-05-06T00:13:03.146 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-06T00:13:03.147 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-06T00:13:03.147 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-06T00:13:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-06T00:13:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-06T00:13:03.155 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-05-06T00:13:03.170 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-06T00:13:03.170 INFO:teuthology.orchestra.run.smithi186.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-06T00:13:03.188 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-06T00:13:03.188 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-06T00:13:03.188 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-06T00:13:03.189 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-06T00:13:03.189 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-06T00:13:03.191 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-06T00:13:03.273 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [620 kB] 2024-05-06T00:13:03.329 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [620 kB] 2024-05-06T00:13:03.475 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,612 kB] 2024-05-06T00:13:03.528 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,612 kB] 2024-05-06T00:13:03.625 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [304 kB] 2024-05-06T00:13:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-06T00:13:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-05-06T00:13:03.682 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [304 kB] 2024-05-06T00:13:03.697 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-06T00:13:03.698 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-05-06T00:13:03.703 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-06T00:13:03.704 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-05-06T00:13:03.712 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-06T00:13:03.760 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-06T00:13:03.761 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-05-06T00:13:03.768 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-06T00:13:03.881 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [701 kB] 2024-05-06T00:13:03.932 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [701 kB] 2024-05-06T00:13:04.469 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,072 kB] 2024-05-06T00:13:04.514 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,072 kB] 2024-05-06T00:13:04.581 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-05-06T00:13:04.602 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-06T00:13:04.603 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-06T00:13:04.606 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-06T00:13:04.607 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-06T00:13:04.608 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-06T00:13:04.608 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-06T00:13:04.611 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-06T00:13:04.616 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-06T00:13:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-05-06T00:13:04.647 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-06T00:13:04.648 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-06T00:13:04.652 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-06T00:13:04.652 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-06T00:13:04.653 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-06T00:13:04.653 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-06T00:13:04.657 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-06T00:13:04.662 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-06T00:13:04.782 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-06T00:13:04.827 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-06T00:13:04.865 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-06T00:13:04.909 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-06T00:13:04.973 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-06T00:13:05.020 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-06T00:13:05.029 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-06T00:13:05.064 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-06T00:13:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-06T00:13:05.104 INFO:teuthology.orchestra.run.smithi078.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-06T00:13:08.766 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 13.3 MB in 2s (7,122 kB/s) 2024-05-06T00:13:09.199 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 13.3 MB in 3s (5,189 kB/s) 2024-05-06T00:13:09.398 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 13.3 MB in 3s (5,201 kB/s) 2024-05-06T00:13:10.082 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-05-06T00:13:10.100 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-06T00:13:10.152 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-05-06T00:13:10.339 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2024-05-06T00:13:10.339 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2024-05-06T00:13:10.499 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:13:10.500 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:13:10.500 INFO:teuthology.orchestra.run.smithi186.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:13:10.500 INFO:teuthology.orchestra.run.smithi186.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:13:10.501 INFO:teuthology.orchestra.run.smithi186.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-06T00:13:10.501 INFO:teuthology.orchestra.run.smithi186.stdout: python2.7-minimal 2024-05-06T00:13:10.501 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:13:10.501 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2024-05-06T00:13:10.502 INFO:teuthology.orchestra.run.smithi186.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-05-06T00:13:10.502 INFO:teuthology.orchestra.run.smithi186.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-05-06T00:13:10.502 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-05-06T00:13:10.504 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2024-05-06T00:13:10.504 INFO:teuthology.orchestra.run.smithi186.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-06T00:13:10.504 INFO:teuthology.orchestra.run.smithi186.stdout:Recommended packages: 2024-05-06T00:13:10.504 INFO:teuthology.orchestra.run.smithi186.stdout: thermald 2024-05-06T00:13:10.556 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2024-05-06T00:13:10.556 INFO:teuthology.orchestra.run.smithi186.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-05-06T00:13:10.556 INFO:teuthology.orchestra.run.smithi186.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-05-06T00:13:10.556 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-extra-5.15.0-105-generic 2024-05-06T00:13:10.557 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages will be upgraded: 2024-05-06T00:13:10.558 INFO:teuthology.orchestra.run.smithi186.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-06T00:13:10.614 INFO:teuthology.orchestra.run.smithi186.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-05-06T00:13:10.615 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 113 MB of archives. 2024-05-06T00:13:10.615 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-06T00:13:10.615 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-05-06T00:13:10.618 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-05-06T00:13:10.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-06T00:13:10.685 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-06T00:13:10.699 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-05-06T00:13:10.702 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-06T00:13:10.761 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-06T00:13:10.900 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-05-06T00:13:10.900 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-05-06T00:13:10.969 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-05-06T00:13:10.970 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-05-06T00:13:11.057 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-05-06T00:13:11.077 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:13:11.077 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:13:11.077 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:13:11.079 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:13:11.079 INFO:teuthology.orchestra.run.smithi146.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-06T00:13:11.079 INFO:teuthology.orchestra.run.smithi146.stdout: python2.7-minimal 2024-05-06T00:13:11.079 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:13:11.080 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2024-05-06T00:13:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-05-06T00:13:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-05-06T00:13:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-05-06T00:13:11.081 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2024-05-06T00:13:11.081 INFO:teuthology.orchestra.run.smithi146.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-06T00:13:11.082 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2024-05-06T00:13:11.082 INFO:teuthology.orchestra.run.smithi146.stdout: thermald 2024-05-06T00:13:11.130 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2024-05-06T00:13:11.130 INFO:teuthology.orchestra.run.smithi146.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-05-06T00:13:11.130 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-05-06T00:13:11.130 INFO:teuthology.orchestra.run.smithi146.stdout: linux-modules-extra-5.15.0-105-generic 2024-05-06T00:13:11.131 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages will be upgraded: 2024-05-06T00:13:11.132 INFO:teuthology.orchestra.run.smithi146.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-06T00:13:11.133 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:13:11.133 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:13:11.133 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:13:11.133 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:13:11.133 INFO:teuthology.orchestra.run.smithi078.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-06T00:13:11.134 INFO:teuthology.orchestra.run.smithi078.stdout: python2.7-minimal 2024-05-06T00:13:11.134 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:13:11.135 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2024-05-06T00:13:11.135 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-05-06T00:13:11.135 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-05-06T00:13:11.135 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-05-06T00:13:11.137 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2024-05-06T00:13:11.137 INFO:teuthology.orchestra.run.smithi078.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-06T00:13:11.137 INFO:teuthology.orchestra.run.smithi078.stdout:Recommended packages: 2024-05-06T00:13:11.137 INFO:teuthology.orchestra.run.smithi078.stdout: thermald 2024-05-06T00:13:11.163 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-05-06T00:13:11.186 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2024-05-06T00:13:11.186 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-05-06T00:13:11.186 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-05-06T00:13:11.186 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-extra-5.15.0-105-generic 2024-05-06T00:13:11.188 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages will be upgraded: 2024-05-06T00:13:11.188 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-06T00:13:11.251 INFO:teuthology.orchestra.run.smithi078.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-05-06T00:13:11.251 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 113 MB of archives. 2024-05-06T00:13:11.251 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-06T00:13:11.251 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-05-06T00:13:11.323 INFO:teuthology.orchestra.run.smithi146.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-05-06T00:13:11.323 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 113 MB of archives. 2024-05-06T00:13:11.323 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-06T00:13:11.323 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-05-06T00:13:11.981 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-05-06T00:13:12.118 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-05-06T00:13:12.819 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-05-06T00:13:12.873 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-05-06T00:13:12.874 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-05-06T00:13:12.874 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-05-06T00:13:13.133 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-05-06T00:13:13.174 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-05-06T00:13:13.177 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-05-06T00:13:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-05-06T00:13:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-05-06T00:13:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-05-06T00:13:13.366 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-05-06T00:13:13.465 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-05-06T00:13:13.565 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 113 MB in 3s (43.4 MB/s) 2024-05-06T00:13:13.761 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-05-06T00:13:13.865 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 113 MB in 2s (49.9 MB/s) 2024-05-06T00:13:14.059 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-05-06T00:13:15.435 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-05-06T00:13:15.435 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-05-06T00:13:15.436 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-05-06T00:13:15.799 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-06T00:13:15.804 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:15.859 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-05-06T00:13:15.980 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-05-06T00:13:16.041 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:16.110 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-06T00:13:16.114 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:16.189 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:16.392 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 113 MB in 5s (23.4 MB/s) 2024-05-06T00:13:16.559 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-05-06T00:13:18.083 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-05-06T00:13:18.105 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:18.199 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:18.306 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-05-06T00:13:18.328 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:18.429 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:18.527 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-06T00:13:18.532 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:18.662 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-05-06T00:13:18.682 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:18.698 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:18.719 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:18.901 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-05-06T00:13:18.922 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:18.959 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:21.084 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-05-06T00:13:21.105 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:21.188 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:21.637 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-05-06T00:13:21.657 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:21.695 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:24.942 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:25.086 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:25.419 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:25.547 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:25.842 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-05-06T00:13:25.864 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-05-06T00:13:25.908 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-05-06T00:13:25.993 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:26.137 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:26.479 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:26.616 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:26.927 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-05-06T00:13:26.950 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-05-06T00:13:26.994 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-05-06T00:13:28.051 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:28.178 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:28.528 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:28.640 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:28.918 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-05-06T00:13:28.940 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-05-06T00:13:28.984 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-05-06T00:13:31.063 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-05-06T00:13:31.100 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:31.138 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:32.910 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-05-06T00:13:32.947 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:32.985 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:34.166 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:34.291 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:34.511 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-05-06T00:13:34.549 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-05-06T00:13:34.586 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:34.625 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-05-06T00:13:34.729 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:34.855 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-05-06T00:13:34.983 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:35.555 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:35.964 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-05-06T00:13:36.061 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:36.195 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-05-06T00:13:36.323 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:36.330 INFO:teuthology.orchestra.run.smithi186.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:13:36.330 INFO:teuthology.orchestra.run.smithi186.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-06T00:13:36.330 INFO:teuthology.orchestra.run.smithi186.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:13:36.330 INFO:teuthology.orchestra.run.smithi186.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:36.485 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:37.096 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-05-06T00:13:37.229 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-05-06T00:13:37.562 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-05-06T00:13:37.633 INFO:teuthology.orchestra.run.smithi078.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:13:37.633 INFO:teuthology.orchestra.run.smithi078.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-06T00:13:37.634 INFO:teuthology.orchestra.run.smithi078.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:13:37.634 INFO:teuthology.orchestra.run.smithi078.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:37.792 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:37.835 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-05-06T00:13:37.920 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:38.197 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:39.179 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:40.014 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-05-06T00:13:40.015 INFO:teuthology.orchestra.run.smithi146.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:13:40.015 INFO:teuthology.orchestra.run.smithi146.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-06T00:13:40.015 INFO:teuthology.orchestra.run.smithi146.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:13:40.015 INFO:teuthology.orchestra.run.smithi146.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:40.140 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-05-06T00:13:40.174 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:40.266 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:40.330 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-06T00:13:40.330 INFO:teuthology.orchestra.run.smithi186.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:40.562 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-05-06T00:13:40.688 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-05-06T00:13:40.814 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:40.882 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-06T00:13:40.883 INFO:teuthology.orchestra.run.smithi078.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:41.609 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:43.030 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-05-06T00:13:43.152 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-05-06T00:13:43.279 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-05-06T00:13:43.348 INFO:teuthology.orchestra.run.smithi146.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-06T00:13:43.348 INFO:teuthology.orchestra.run.smithi146.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:54.724 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-06T00:13:54.724 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub' 2024-05-06T00:13:54.740 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-06T00:13:54.760 INFO:teuthology.orchestra.run.smithi186.stdout:Generating grub configuration file ... 2024-05-06T00:13:54.928 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-06T00:13:54.928 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub' 2024-05-06T00:13:54.938 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-06T00:13:54.957 INFO:teuthology.orchestra.run.smithi078.stdout:Generating grub configuration file ... 2024-05-06T00:13:55.009 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:13:55.025 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:55.224 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:13:55.241 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:55.286 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:13:55.292 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-06T00:13:55.360 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-06T00:13:55.365 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-06T00:13:55.523 INFO:teuthology.orchestra.run.smithi186.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-06T00:13:55.524 INFO:teuthology.orchestra.run.smithi186.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-06T00:13:55.524 INFO:teuthology.orchestra.run.smithi186.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-06T00:13:55.564 INFO:teuthology.orchestra.run.smithi186.stdout:done 2024-05-06T00:13:55.567 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:13:55.573 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-06T00:13:55.646 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-06T00:13:55.652 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-06T00:13:55.841 INFO:teuthology.orchestra.run.smithi078.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-06T00:13:55.841 INFO:teuthology.orchestra.run.smithi078.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-06T00:13:55.841 INFO:teuthology.orchestra.run.smithi078.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-06T00:13:55.883 INFO:teuthology.orchestra.run.smithi078.stdout:done 2024-05-06T00:13:56.013 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic 2024-05-06T00:13:56.039 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic 2024-05-06T00:13:56.039 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2024-05-06T00:13:56.039 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2024-05-06T00:13:56.039 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2024-05-06T00:13:56.039 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 21 2024-05-06T00:13:56.039 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2024-05-06T00:13:56.039 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2024-05-06T00:13:56.039 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta 2024-05-06T00:13:56.040 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.15.0.105.102 2024-05-06T00:13:56.040 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-06T00:13:56.040 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-06T00:13:56.040 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2024-05-06T00:13:56.040 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2024-05-06T00:13:56.040 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2024-05-06T00:13:56.040 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2024-05-06T00:13:56.040 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-05-06T00:13:56.040 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-06T00:13:56.040 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-06T00:13:56.041 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-06T00:13:56.041 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-05-06T00:13:56.041 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2024-05-06T00:13:56.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2024-05-06T00:13:56.326 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2024-05-06T00:13:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2024-05-06T00:13:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2024-05-06T00:13:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2024-05-06T00:13:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2024-05-06T00:13:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 21 2024-05-06T00:13:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2024-05-06T00:13:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2024-05-06T00:13:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2024-05-06T00:13:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.15.0.105.102 2024-05-06T00:13:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-06T00:13:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-06T00:13:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2024-05-06T00:13:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2024-05-06T00:13:56.344 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2024-05-06T00:13:56.344 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2024-05-06T00:13:56.344 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-05-06T00:13:56.344 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-06T00:13:56.344 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-06T00:13:56.344 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-06T00:13:56.345 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-05-06T00:13:56.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2024-05-06T00:13:56.355 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-06T00:13:56.484 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:13:56.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-05-06T00:13:56.554 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-06T00:13:56.637 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-06T00:13:56.680 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-06T00:13:56.791 INFO:teuthology.orchestra.run.smithi146.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-06T00:13:56.791 INFO:teuthology.orchestra.run.smithi146.stdout:Sourcing file `/etc/default/grub' 2024-05-06T00:13:56.793 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:13:56.797 INFO:teuthology.orchestra.run.smithi146.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-06T00:13:56.818 INFO:teuthology.orchestra.run.smithi146.stdout:Generating grub configuration file ... 2024-05-06T00:13:56.889 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-06T00:13:56.955 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-06T00:13:57.085 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:13:57.100 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:57.388 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:13:57.394 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-06T00:13:57.464 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-06T00:13:57.470 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-06T00:13:57.646 INFO:teuthology.orchestra.run.smithi146.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-06T00:13:57.647 INFO:teuthology.orchestra.run.smithi146.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-06T00:13:57.647 INFO:teuthology.orchestra.run.smithi146.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-06T00:13:57.695 INFO:teuthology.orchestra.run.smithi146.stdout:done 2024-05-06T00:13:57.729 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-05-06T00:13:57.745 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-06T00:13:57.802 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-05-06T00:13:57.997 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2024-05-06T00:13:57.997 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2024-05-06T00:13:58.093 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-06T00:13:58.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-06T00:13:58.160 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2024-05-06T00:13:58.169 INFO:teuthology.orchestra.run.smithi186.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-05-06T00:13:58.169 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:13:58.169 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:13:58.169 INFO:teuthology.orchestra.run.smithi186.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:13:58.169 INFO:teuthology.orchestra.run.smithi186.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:13:58.169 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-06T00:13:58.170 INFO:teuthology.orchestra.run.smithi186.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-06T00:13:58.170 INFO:teuthology.orchestra.run.smithi186.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-06T00:13:58.170 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 21 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.15.0.105.102 2024-05-06T00:13:58.179 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-06T00:13:58.180 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-06T00:13:58.180 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2024-05-06T00:13:58.180 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2024-05-06T00:13:58.180 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2024-05-06T00:13:58.180 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2024-05-06T00:13:58.181 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-05-06T00:13:58.181 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-06T00:13:58.181 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-06T00:13:58.181 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-06T00:13:58.181 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-05-06T00:13:58.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2024-05-06T00:13:58.210 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-05-06T00:13:58.212 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 21 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta 2024-05-06T00:13:58.228 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.15.0.105.102 2024-05-06T00:13:58.229 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-06T00:13:58.229 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-06T00:13:58.229 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2024-05-06T00:13:58.229 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2024-05-06T00:13:58.229 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2024-05-06T00:13:58.229 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2024-05-06T00:13:58.229 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-05-06T00:13:58.273 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.URoBmqlHEG 2024-05-06T00:13:58.274 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub/grub.cfg /tmp/tmp.URoBmqlHEG 2024-05-06T00:13:58.328 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.URoBmqlHEG 2024-05-06T00:13:58.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-05-06T00:13:58.367 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-05-06T00:13:58.368 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-05-06T00:13:58.443 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.URoBmqlHEG is 10KB 2024-05-06T00:13:58.455 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.URoBmqlHEG 2024-05-06T00:13:58.460 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-06T00:13:58.492 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:13:58.500 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-06T00:13:58.514 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:13:58.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-06T00:13:58.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-06T00:13:58.520 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-06T00:13:58.536 INFO:teuthology.orchestra.run.smithi078.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-05-06T00:13:58.536 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:13:58.536 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:13:58.536 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:13:58.536 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:13:58.537 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-06T00:13:58.537 INFO:teuthology.orchestra.run.smithi078.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-06T00:13:58.537 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:13:58.576 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-05-06T00:13:58.578 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2024-05-06T00:13:58.589 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-05-06T00:13:58.589 DEBUG:teuthology.orchestra.run.smithi186:> sudo update-grub 2024-05-06T00:13:58.592 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 21 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.15.0.105.102 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2024-05-06T00:13:58.593 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2024-05-06T00:13:58.594 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2024-05-06T00:13:58.594 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-05-06T00:13:58.609 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-06T00:13:58.638 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.p48jQgPnFW 2024-05-06T00:13:58.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub/grub.cfg /tmp/tmp.p48jQgPnFW 2024-05-06T00:13:58.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.p48jQgPnFW 2024-05-06T00:13:58.806 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.p48jQgPnFW is 10KB 2024-05-06T00:13:58.818 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.p48jQgPnFW 2024-05-06T00:13:58.824 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-06T00:13:58.878 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:13:58.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-06T00:13:58.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-06T00:13:58.958 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-05-06T00:13:58.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo update-grub 2024-05-06T00:13:59.694 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub' 2024-05-06T00:13:59.695 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-06T00:13:59.699 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-05-06T00:13:59.816 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-05-06T00:13:59.835 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-06T00:13:59.877 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:13:59.883 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:13:59.885 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-05-06T00:14:00.095 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-05-06T00:14:00.095 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-05-06T00:14:00.116 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:14:00.121 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-06T00:14:00.163 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-06T00:14:00.168 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-06T00:14:00.200 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub' 2024-05-06T00:14:00.201 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-06T00:14:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-05-06T00:14:00.251 INFO:teuthology.orchestra.run.smithi146.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-05-06T00:14:00.251 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:14:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:14:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:14:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:14:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-06T00:14:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-06T00:14:00.252 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:14:00.289 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-05-06T00:14:00.291 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2024-05-06T00:14:00.292 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-06T00:14:00.292 INFO:teuthology.orchestra.run.smithi186.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-06T00:14:00.292 INFO:teuthology.orchestra.run.smithi186.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-06T00:14:00.302 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-05-06T00:14:00.315 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 21 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2024-05-06T00:14:00.355 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.15.0.105.102 2024-05-06T00:14:00.356 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-06T00:14:00.356 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-06T00:14:00.356 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2024-05-06T00:14:00.356 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2024-05-06T00:14:00.356 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2024-05-06T00:14:00.356 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2024-05-06T00:14:00.356 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-05-06T00:14:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:14:00.400 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.zX3PKYnW7H 2024-05-06T00:14:00.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub/grub.cfg /tmp/tmp.zX3PKYnW7H 2024-05-06T00:14:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:14:00.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.zX3PKYnW7H 2024-05-06T00:14:00.575 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.zX3PKYnW7H is 10KB 2024-05-06T00:14:00.587 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.zX3PKYnW7H 2024-05-06T00:14:00.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-06T00:14:00.633 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:14:00.639 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-06T00:14:00.647 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:14:00.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-06T00:14:00.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-06T00:14:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-06T00:14:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-06T00:14:00.714 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-05-06T00:14:00.714 DEBUG:teuthology.orchestra.run.smithi146:> sudo update-grub 2024-05-06T00:14:00.801 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-06T00:14:00.801 INFO:teuthology.orchestra.run.smithi078.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-06T00:14:00.801 INFO:teuthology.orchestra.run.smithi078.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-06T00:14:00.816 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-05-06T00:14:00.819 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-05-06T00:14:01.866 INFO:teuthology.orchestra.run.smithi146.stderr:Sourcing file `/etc/default/grub' 2024-05-06T00:14:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-06T00:14:01.869 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-05-06T00:14:02.056 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-05-06T00:14:02.061 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-05-06T00:14:02.288 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-06T00:14:02.294 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-06T00:14:02.337 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-06T00:14:02.342 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-06T00:14:02.481 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-06T00:14:02.481 INFO:teuthology.orchestra.run.smithi146.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-06T00:14:02.481 INFO:teuthology.orchestra.run.smithi146.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-06T00:14:02.498 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-05-06T00:14:02.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-05-06T00:14:30.329 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-06T00:14:30.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-06T00:14:30.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:14:30.822 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-06T00:14:30.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-06T00:14:30.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:14:32.505 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-06T00:14:32.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-06T00:14:32.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:14:48.863 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-05-06T00:14:49.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-05-06T00:14:50.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-06T00:14:57.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-06T00:14:57.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:14:58.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-06T00:14:58.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:14:59.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-06T00:14:59.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:15:00.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-05-06T00:15:01.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-05-06T00:15:02.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-06T00:15:12.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-06T00:15:12.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:15:13.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-06T00:15:13.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:15:14.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-06T00:15:14.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:15:31.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-05-06T00:15:46.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-06T00:15:46.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:15:47.565 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-05-06T00:15:48.436 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-06T00:15:48.437 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-05-06T00:15:48.437 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-05-06T00:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout:5.15.0-105-generic 2024-05-06T00:15:48.483 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-05-06T00:15:48.483 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-06T00:15:48.483 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-06T00:15:49.483 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-06T00:15:49.484 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T00:15:49.556 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-05-06T00:15:49.584 DEBUG:teuthology.parallel:result is None 2024-05-06T00:16:12.941 DEBUG:teuthology.orchestra.remote:timed out 2024-05-06T00:16:14.978 DEBUG:teuthology.orchestra.remote:timed out 2024-05-06T00:16:27.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-06T00:16:27.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:16:28.273 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-05-06T00:16:28.812 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-06T00:16:28.813 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.15.0-105-generic"... 2024-05-06T00:16:28.813 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-05-06T00:16:28.859 INFO:teuthology.orchestra.run.smithi186.stdout:5.15.0-105-generic 2024-05-06T00:16:28.859 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-05-06T00:16:28.859 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-06T00:16:28.860 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-06T00:16:29.860 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-05-06T00:16:29.861 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T00:16:29.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-06T00:16:29.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:16:30.202 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-05-06T00:16:30.237 DEBUG:teuthology.parallel:result is None 2024-05-06T00:16:30.266 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-05-06T00:16:30.851 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-06T00:16:30.851 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-05-06T00:16:30.851 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-05-06T00:16:30.898 INFO:teuthology.orchestra.run.smithi146.stdout:5.15.0-105-generic 2024-05-06T00:16:30.898 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-05-06T00:16:30.898 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-06T00:16:30.898 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-06T00:16:31.899 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-06T00:16:31.899 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T00:16:31.981 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-05-06T00:16:32.000 DEBUG:teuthology.parallel:result is None 2024-05-06T00:16:32.000 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-06T00:16:32.008 INFO:teuthology.task.internal:Creating test directory... 2024-05-06T00:16:32.008 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T00:16:32.011 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T00:16:32.014 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T00:16:32.020 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-06T00:16:32.040 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-06T00:16:32.048 INFO:teuthology.task.internal:Creating archive directory... 2024-05-06T00:16:32.049 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T00:16:32.060 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T00:16:32.063 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T00:16:32.083 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-06T00:16:32.089 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-06T00:16:32.090 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T00:16:32.120 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T00:16:32.122 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T00:16:32.141 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T00:16:32.142 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T00:16:32.143 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T00:16:32.150 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T00:16:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T00:16:32.152 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T00:16:32.153 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-06T00:16:32.160 INFO:teuthology.task.internal:Configuring sudo... 2024-05-06T00:16:32.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T00:16:32.195 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T00:16:32.198 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T00:16:32.217 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-06T00:16:32.229 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-06T00:16:32.229 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T00:16:32.252 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T00:16:32.259 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T00:16:32.264 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T00:16:32.302 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T00:16:32.351 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:16:32.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T00:16:32.415 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T00:16:32.421 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T00:16:32.467 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:16:32.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T00:16:32.537 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T00:16:32.544 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T00:16:32.592 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:16:32.592 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T00:16:32.661 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-05-06T00:16:32.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-05-06T00:16:32.666 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-05-06T00:16:32.738 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-06T00:16:32.745 INFO:teuthology.task.internal:Starting timer... 2024-05-06T00:16:32.746 INFO:teuthology.run_tasks:Running task pcp... 2024-05-06T00:16:32.768 INFO:teuthology.run_tasks:Running task selinux... 2024-05-06T00:16:32.777 DEBUG:teuthology.task.selinux:Excluding smithi078: OS 'ubuntu' does not support SELinux 2024-05-06T00:16:32.777 DEBUG:teuthology.task.selinux:Excluding smithi146: OS 'ubuntu' does not support SELinux 2024-05-06T00:16:32.777 DEBUG:teuthology.task.selinux:Excluding smithi186: OS 'ubuntu' does not support SELinux 2024-05-06T00:16:32.777 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-06T00:16:32.777 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-05-06T00:16:32.778 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-06T00:16:32.778 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-06T00:16:32.786 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-06T00:16:32.792 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-06T00:16:32.930 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-06T00:16:32.944 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-06T00:16:32.946 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi146.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-06T00:23:36.614 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-05-06T00:23:36.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-06T00:23:36.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:23:36.701 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-05-06T00:23:36.774 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-06T00:23:36.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-06T00:23:36.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:23:36.853 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-05-06T00:23:36.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-06T00:23:36.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-06T00:23:36.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T00:23:37.006 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-05-06T00:23:37.082 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-06T00:23:37.082 INFO:teuthology.run_tasks:Running task clock... 2024-05-06T00:23:37.093 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-06T00:23:37.093 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T00:23:37.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T00:23:37.096 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T00:23:37.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T00:23:37.098 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T00:23:37.098 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T00:23:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-06T00:23:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Command line: ntpd -gq 2024-05-06T00:23:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: ---------------------------------------------------- 2024-05-06T00:23:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: ntp-4 is maintained by Network Time Foundation, 2024-05-06T00:23:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-06T00:23:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: corporation. Support and training for ntp-4 are 2024-05-06T00:23:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: available at https://www.nwtime.org/support 2024-05-06T00:23:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: ---------------------------------------------------- 2024-05-06T00:23:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: proto: precision = 0.060 usec (-24) 2024-05-06T00:23:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: basedate set to 2022-02-04 2024-05-06T00:23:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: gps base set to 2022-02-06 (week 2196) 2024-05-06T00:23:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-06T00:23:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-06T00:23:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-06T00:23:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: restrict ::: KOD does nothing without LIMITED. 2024-05-06T00:23:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Listen and drop on 0 v6wildcard [::]:123 2024-05-06T00:23:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-06T00:23:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Listen normally on 2 lo 127.0.0.1:123 2024-05-06T00:23:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Listen normally on 3 enp3s0f1 172.21.15.78:123 2024-05-06T00:23:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Listen normally on 4 lo [::1]:123 2024-05-06T00:23:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d4dd%5]:123 2024-05-06T00:23:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:37 ntpd[17245]: Listening on routing socket on fd #22 for interface updates 2024-05-06T00:23:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-06T00:23:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Command line: ntpd -gq 2024-05-06T00:23:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: ---------------------------------------------------- 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: ntp-4 is maintained by Network Time Foundation, 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: corporation. Support and training for ntp-4 are 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: available at https://www.nwtime.org/support 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: ---------------------------------------------------- 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: proto: precision = 0.057 usec (-24) 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: basedate set to 2022-02-04 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: gps base set to 2022-02-06 (week 2196) 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-06T00:23:37.126 INFO:teuthology.orchestra.run.smithi146.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-06T00:23:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: restrict ::: KOD does nothing without LIMITED. 2024-05-06T00:23:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Listen and drop on 0 v6wildcard [::]:123 2024-05-06T00:23:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-06T00:23:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Listen normally on 2 lo 127.0.0.1:123 2024-05-06T00:23:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Listen normally on 3 enp3s0f1 172.21.15.146:123 2024-05-06T00:23:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Listen normally on 4 lo [::1]:123 2024-05-06T00:23:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:76fd%5]:123 2024-05-06T00:23:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:37 ntpd[17265]: Listening on routing socket on fd #22 for interface updates 2024-05-06T00:23:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-06T00:23:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Command line: ntpd -gq 2024-05-06T00:23:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: ---------------------------------------------------- 2024-05-06T00:23:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: ntp-4 is maintained by Network Time Foundation, 2024-05-06T00:23:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-06T00:23:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: corporation. Support and training for ntp-4 are 2024-05-06T00:23:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: available at https://www.nwtime.org/support 2024-05-06T00:23:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: ---------------------------------------------------- 2024-05-06T00:23:37.157 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: proto: precision = 0.060 usec (-24) 2024-05-06T00:23:37.157 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: basedate set to 2022-02-04 2024-05-06T00:23:37.157 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: gps base set to 2022-02-06 (week 2196) 2024-05-06T00:23:37.157 INFO:teuthology.orchestra.run.smithi186.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-06T00:23:37.158 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-06T00:23:37.158 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: restrict ::: KOD does nothing without LIMITED. 2024-05-06T00:23:37.158 INFO:teuthology.orchestra.run.smithi186.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-06T00:23:37.159 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Listen and drop on 0 v6wildcard [::]:123 2024-05-06T00:23:37.159 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-06T00:23:37.159 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Listen normally on 2 lo 127.0.0.1:123 2024-05-06T00:23:37.159 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Listen normally on 3 enp3s0f1 172.21.15.186:123 2024-05-06T00:23:37.159 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Listen normally on 4 lo [::1]:123 2024-05-06T00:23:37.159 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df5%5]:123 2024-05-06T00:23:37.159 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:37 ntpd[17248]: Listening on routing socket on fd #22 for interface updates 2024-05-06T00:23:38.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:38 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:38.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:38 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:38.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:38 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:38.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:38 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:38.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:38 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:38.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:38 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:38.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:38 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:38.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:38 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:38.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:38 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:38.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:38 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:38.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:38 ntpd[17248]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:38.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:38 ntpd[17248]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:39.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:39 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:39.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:39 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:39.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:39 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:39.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:39 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:39.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:39 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:39.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:39 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:39.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:39 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:39.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:39 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:39.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:39 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:39.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:39 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:40.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:40 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:40.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:40 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:40.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:40 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:40.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:40 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:40.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:40 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:40.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:40 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:40.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:40 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:40.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:40 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:40.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:40 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:40.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:40 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:40.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:40 ntpd[17248]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:40.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:40 ntpd[17248]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:41.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:41.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:41 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:41.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:41 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:41 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:41 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:41 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:41 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:41 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:41 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:41.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:41 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:41.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:41 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:41 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:41 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:41.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:41 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:41.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:41 ntpd[17248]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:41.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:41 ntpd[17248]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:42.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:42 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:42 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:42 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:42 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:42 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:42.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:42 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:42.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:42.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:42 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:42.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:42.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:42.158 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:42.158 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:42.158 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:42.158 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:42 ntpd[17248]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:43.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:43.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:43 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:43.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:43 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:43.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:43 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:43.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:43 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:43.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:43 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:43.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:43 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:43.124 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:43 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:43.156 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:43 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:43.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:43 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:43.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:43 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:43.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:43 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:43.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:43 ntpd[17248]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:43.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:43 ntpd[17248]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:44.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:44 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:44 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:44 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:44 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:44 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:44.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:44 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.125 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:44.126 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: 6 May 00:23:44 ntpd[17265]: ntpd: time slew +0.000783 s 2024-05-06T00:23:44.126 INFO:teuthology.orchestra.run.smithi146.stdout:ntpd: time slew +0.000783s 2024-05-06T00:23:44.126 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-06T00:23:44.127 INFO:teuthology.orchestra.run.smithi146.stderr: 6 May 00:23:44 ntpd[17265]: can't open /var/log/ntpstats/loopstats.20240506: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:44.157 INFO:teuthology.orchestra.run.smithi186.stdout: 6 May 00:23:44 ntpd[17248]: ntpd: time slew -0.000208 s 2024-05-06T00:23:44.158 INFO:teuthology.orchestra.run.smithi186.stdout:ntpd: time slew -0.000208s 2024-05-06T00:23:44.158 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-06T00:23:44.158 INFO:teuthology.orchestra.run.smithi186.stderr: 6 May 00:23:44 ntpd[17248]: can't open /var/log/ntpstats/loopstats.20240506: Permission denied 2024-05-06T00:23:44.183 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2024-05-06T00:23:44.183 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2024-05-06T00:23:44.183 INFO:teuthology.orchestra.run.smithi146.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:44.183 INFO:teuthology.orchestra.run.smithi146.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:44.183 INFO:teuthology.orchestra.run.smithi146.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:44.183 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:44.204 INFO:teuthology.orchestra.run.smithi186.stdout: remote refid st t when poll reach delay offset jitter 2024-05-06T00:23:44.204 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================== 2024-05-06T00:23:44.204 INFO:teuthology.orchestra.run.smithi186.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:44.204 INFO:teuthology.orchestra.run.smithi186.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:44.204 INFO:teuthology.orchestra.run.smithi186.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:44.204 INFO:teuthology.orchestra.run.smithi186.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:45.122 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: can't open /var/log/ntpstats/rawstats.20240506: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-06T00:23:45.123 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: can't open /var/log/ntpstats/peerstats.20240506: Permission denied 2024-05-06T00:23:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: 6 May 00:23:45 ntpd[17245]: ntpd: time slew -0.000787 s 2024-05-06T00:23:45.124 INFO:teuthology.orchestra.run.smithi078.stdout:ntpd: time slew -0.000787s 2024-05-06T00:23:45.124 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-06T00:23:45.124 INFO:teuthology.orchestra.run.smithi078.stderr: 6 May 00:23:45 ntpd[17245]: can't open /var/log/ntpstats/loopstats.20240506: Permission denied 2024-05-06T00:23:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2024-05-06T00:23:45.173 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2024-05-06T00:23:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:23:45.174 INFO:teuthology.run_tasks:Running task install... 2024-05-06T00:23:45.183 DEBUG:teuthology.task.install:project ceph 2024-05-06T00:23:45.184 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'b45ed4400464709677363de54673bcce970bcb27'}} 2024-05-06T00:23:45.184 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'b45ed4400464709677363de54673bcce970bcb27'} 2024-05-06T00:23:45.184 INFO:teuthology.task.install:Using flavor: default 2024-05-06T00:23:45.189 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-06T00:23:45.189 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-05-06T00:23:45.190 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-key list | grep Ceph 2024-05-06T00:23:45.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-key list | grep Ceph 2024-05-06T00:23:45.191 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-key list | grep Ceph 2024-05-06T00:23:45.254 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-06T00:23:45.256 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-06T00:23:45.278 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-06T00:23:45.282 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-06T00:23:45.282 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph.com (release key) 2024-05-06T00:23:45.283 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-05-06T00:23:45.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b45ed4400464709677363de54673bcce970bcb27 2024-05-06T00:23:45.289 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-06T00:23:45.289 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph.com (release key) 2024-05-06T00:23:45.290 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-05-06T00:23:45.290 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b45ed4400464709677363de54673bcce970bcb27 2024-05-06T00:23:45.305 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-06T00:23:45.305 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph.com (release key) 2024-05-06T00:23:45.306 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-05-06T00:23:45.307 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b45ed4400464709677363de54673bcce970bcb27 2024-05-06T00:23:45.431 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default/ 2024-05-06T00:23:45.431 INFO:teuthology.task.install.deb:Package version is 19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:23:45.454 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default/ 2024-05-06T00:23:45.455 INFO:teuthology.task.install.deb:Package version is 19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:23:45.500 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default/ 2024-05-06T00:23:45.500 INFO:teuthology.task.install.deb:Package version is 19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:23:45.567 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:23:45.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-06T00:23:45.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-05-06T00:23:45.599 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:23:45.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-06T00:23:45.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-05-06T00:23:45.670 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:23:45.670 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-06T00:23:45.684 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2024-05-06T00:23:45.759 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-06T00:23:45.784 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-06T00:23:45.784 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:23:45.799 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-06T00:23:45.805 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy InRelease 2024-05-06T00:23:45.819 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-06T00:23:45.826 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-06T00:23:45.834 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy InRelease 2024-05-06T00:23:45.848 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-06T00:23:45.856 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-06T00:23:45.856 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-06T00:23:45.856 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:23:45.869 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-05-06T00:23:45.871 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-06T00:23:45.877 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-06T00:23:45.881 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-06T00:23:45.892 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-06T00:23:45.896 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2024-05-06T00:23:45.898 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2024-05-06T00:23:45.905 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy InRelease 2024-05-06T00:23:45.926 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-06T00:23:45.930 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-05-06T00:23:45.948 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-06T00:23:45.963 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-06T00:23:45.969 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-05-06T00:23:46.000 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2024-05-06T00:23:46.045 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-06T00:23:46.141 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 29.2 kB in 0s (84.1 kB/s) 2024-05-06T00:23:46.178 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 29.2 kB in 0s (70.1 kB/s) 2024-05-06T00:23:46.188 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 29.2 kB in 0s (88.8 kB/s) 2024-05-06T00:23:47.117 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-05-06T00:23:47.134 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3517-gb45ed440-1jammy cephadm=19.0.0-3517-gb45ed440-1jammy ceph-mds=19.0.0-3517-gb45ed440-1jammy ceph-mgr=19.0.0-3517-gb45ed440-1jammy ceph-common=19.0.0-3517-gb45ed440-1jammy ceph-fuse=19.0.0-3517-gb45ed440-1jammy ceph-test=19.0.0-3517-gb45ed440-1jammy ceph-volume=19.0.0-3517-gb45ed440-1jammy radosgw=19.0.0-3517-gb45ed440-1jammy python3-rados=19.0.0-3517-gb45ed440-1jammy python3-rgw=19.0.0-3517-gb45ed440-1jammy python3-cephfs=19.0.0-3517-gb45ed440-1jammy python3-rbd=19.0.0-3517-gb45ed440-1jammy libcephfs2=19.0.0-3517-gb45ed440-1jammy libcephfs-dev=19.0.0-3517-gb45ed440-1jammy librados2=19.0.0-3517-gb45ed440-1jammy librbd1=19.0.0-3517-gb45ed440-1jammy rbd-fuse=19.0.0-3517-gb45ed440-1jammy rbd-nbd=19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:23:47.143 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-05-06T00:23:47.152 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-06T00:23:47.159 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3517-gb45ed440-1jammy cephadm=19.0.0-3517-gb45ed440-1jammy ceph-mds=19.0.0-3517-gb45ed440-1jammy ceph-mgr=19.0.0-3517-gb45ed440-1jammy ceph-common=19.0.0-3517-gb45ed440-1jammy ceph-fuse=19.0.0-3517-gb45ed440-1jammy ceph-test=19.0.0-3517-gb45ed440-1jammy ceph-volume=19.0.0-3517-gb45ed440-1jammy radosgw=19.0.0-3517-gb45ed440-1jammy python3-rados=19.0.0-3517-gb45ed440-1jammy python3-rgw=19.0.0-3517-gb45ed440-1jammy python3-cephfs=19.0.0-3517-gb45ed440-1jammy python3-rbd=19.0.0-3517-gb45ed440-1jammy libcephfs2=19.0.0-3517-gb45ed440-1jammy libcephfs-dev=19.0.0-3517-gb45ed440-1jammy librados2=19.0.0-3517-gb45ed440-1jammy librbd1=19.0.0-3517-gb45ed440-1jammy rbd-fuse=19.0.0-3517-gb45ed440-1jammy rbd-nbd=19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:23:47.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3517-gb45ed440-1jammy cephadm=19.0.0-3517-gb45ed440-1jammy ceph-mds=19.0.0-3517-gb45ed440-1jammy ceph-mgr=19.0.0-3517-gb45ed440-1jammy ceph-common=19.0.0-3517-gb45ed440-1jammy ceph-fuse=19.0.0-3517-gb45ed440-1jammy ceph-test=19.0.0-3517-gb45ed440-1jammy ceph-volume=19.0.0-3517-gb45ed440-1jammy radosgw=19.0.0-3517-gb45ed440-1jammy python3-rados=19.0.0-3517-gb45ed440-1jammy python3-rgw=19.0.0-3517-gb45ed440-1jammy python3-cephfs=19.0.0-3517-gb45ed440-1jammy python3-rbd=19.0.0-3517-gb45ed440-1jammy libcephfs2=19.0.0-3517-gb45ed440-1jammy libcephfs-dev=19.0.0-3517-gb45ed440-1jammy librados2=19.0.0-3517-gb45ed440-1jammy librbd1=19.0.0-3517-gb45ed440-1jammy rbd-fuse=19.0.0-3517-gb45ed440-1jammy rbd-nbd=19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:23:47.185 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-05-06T00:23:47.219 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-05-06T00:23:47.220 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-06T00:23:47.370 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-05-06T00:23:47.370 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-05-06T00:23:47.396 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-05-06T00:23:47.397 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-05-06T00:23:47.414 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2024-05-06T00:23:47.415 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2024-05-06T00:23:47.552 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:23:47.553 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:23:47.553 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:23:47.553 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:23:47.553 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-06T00:23:47.553 INFO:teuthology.orchestra.run.smithi146.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-06T00:23:47.553 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:23:47.555 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2024-05-06T00:23:47.555 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-06T00:23:47.555 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-06T00:23:47.555 INFO:teuthology.orchestra.run.smithi146.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-06T00:23:47.555 INFO:teuthology.orchestra.run.smithi146.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-06T00:23:47.556 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet zip 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-06T00:23:47.557 INFO:teuthology.orchestra.run.smithi146.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-06T00:23:47.558 INFO:teuthology.orchestra.run.smithi146.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-06T00:23:47.558 INFO:teuthology.orchestra.run.smithi146.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-06T00:23:47.558 INFO:teuthology.orchestra.run.smithi146.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-06T00:23:47.558 INFO:teuthology.orchestra.run.smithi146.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-06T00:23:47.558 INFO:teuthology.orchestra.run.smithi146.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-06T00:23:47.558 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2024-05-06T00:23:47.558 INFO:teuthology.orchestra.run.smithi146.stdout: btrfs-tools 2024-05-06T00:23:47.567 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:23:47.567 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:23:47.567 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:23:47.567 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:23:47.568 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-06T00:23:47.568 INFO:teuthology.orchestra.run.smithi078.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-06T00:23:47.568 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:23:47.569 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2024-05-06T00:23:47.569 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-06T00:23:47.569 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-06T00:23:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-05-06T00:23:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet zip 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout:Recommended packages: 2024-05-06T00:23:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: btrfs-tools 2024-05-06T00:23:47.609 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2024-05-06T00:23:47.609 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-06T00:23:47.609 INFO:teuthology.orchestra.run.smithi186.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-06T00:23:47.609 INFO:teuthology.orchestra.run.smithi186.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-06T00:23:47.610 INFO:teuthology.orchestra.run.smithi186.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-06T00:23:47.610 INFO:teuthology.orchestra.run.smithi186.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-06T00:23:47.610 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-06T00:23:47.611 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2024-05-06T00:23:47.611 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-06T00:23:47.611 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-06T00:23:47.611 INFO:teuthology.orchestra.run.smithi186.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-06T00:23:47.611 INFO:teuthology.orchestra.run.smithi186.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-06T00:23:47.612 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-06T00:23:47.612 INFO:teuthology.orchestra.run.smithi186.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-06T00:23:47.612 INFO:teuthology.orchestra.run.smithi186.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-06T00:23:47.612 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-06T00:23:47.613 INFO:teuthology.orchestra.run.smithi186.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-06T00:23:47.614 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-06T00:23:47.614 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-05-06T00:23:47.614 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet zip 2024-05-06T00:23:47.614 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2024-05-06T00:23:47.614 INFO:teuthology.orchestra.run.smithi186.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-06T00:23:47.614 INFO:teuthology.orchestra.run.smithi186.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-06T00:23:47.615 INFO:teuthology.orchestra.run.smithi186.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-06T00:23:47.615 INFO:teuthology.orchestra.run.smithi186.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-06T00:23:47.615 INFO:teuthology.orchestra.run.smithi186.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-06T00:23:47.615 INFO:teuthology.orchestra.run.smithi186.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-06T00:23:47.615 INFO:teuthology.orchestra.run.smithi186.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-06T00:23:47.615 INFO:teuthology.orchestra.run.smithi186.stdout:Recommended packages: 2024-05-06T00:23:47.615 INFO:teuthology.orchestra.run.smithi186.stdout: btrfs-tools 2024-05-06T00:23:47.658 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2024-05-06T00:23:47.658 INFO:teuthology.orchestra.run.smithi146.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-06T00:23:47.658 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-06T00:23:47.658 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-06T00:23:47.658 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-06T00:23:47.658 INFO:teuthology.orchestra.run.smithi146.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-06T00:23:47.658 INFO:teuthology.orchestra.run.smithi146.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-06T00:23:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-06T00:23:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-05-06T00:23:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-05-06T00:23:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-05-06T00:23:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-05-06T00:23:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-05-06T00:23:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-05-06T00:23:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: radosgw rbd-fuse rbd-nbd socat unzip xmlstarlet zip 2024-05-06T00:23:47.672 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2024-05-06T00:23:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-06T00:23:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-06T00:23:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-06T00:23:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-06T00:23:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-06T00:23:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-06T00:23:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-06T00:23:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-06T00:23:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-05-06T00:23:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-05-06T00:23:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-05-06T00:23:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-06T00:23:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-05-06T00:23:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: radosgw rbd-fuse rbd-nbd socat unzip xmlstarlet zip 2024-05-06T00:23:47.715 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2024-05-06T00:23:47.715 INFO:teuthology.orchestra.run.smithi186.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-06T00:23:47.715 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-06T00:23:47.715 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-06T00:23:47.715 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-06T00:23:47.715 INFO:teuthology.orchestra.run.smithi186.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-06T00:23:47.715 INFO:teuthology.orchestra.run.smithi186.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-06T00:23:47.716 INFO:teuthology.orchestra.run.smithi186.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-05-06T00:23:47.717 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-05-06T00:23:47.718 INFO:teuthology.orchestra.run.smithi186.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-05-06T00:23:47.718 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-05-06T00:23:47.718 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-05-06T00:23:47.718 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-05-06T00:23:47.718 INFO:teuthology.orchestra.run.smithi186.stdout: radosgw rbd-fuse rbd-nbd socat unzip xmlstarlet zip 2024-05-06T00:23:47.739 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 119 newly installed, 0 to remove and 303 not upgraded. 2024-05-06T00:23:47.739 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 181 MB of archives. 2024-05-06T00:23:47.739 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 829 MB of additional disk space will be used. 2024-05-06T00:23:47.739 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-06T00:23:47.779 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 119 newly installed, 0 to remove and 303 not upgraded. 2024-05-06T00:23:47.779 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 181 MB of archives. 2024-05-06T00:23:47.779 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 829 MB of additional disk space will be used. 2024-05-06T00:23:47.779 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-06T00:23:47.816 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3517-gb45ed440-1jammy [3,555 kB] 2024-05-06T00:23:47.846 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-06T00:23:47.848 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 119 newly installed, 0 to remove and 303 not upgraded. 2024-05-06T00:23:47.848 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 181 MB of archives. 2024-05-06T00:23:47.848 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 829 MB of additional disk space will be used. 2024-05-06T00:23:47.848 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-06T00:23:47.849 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-06T00:23:47.869 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-06T00:23:47.879 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3517-gb45ed440-1jammy [3,555 kB] 2024-05-06T00:23:47.881 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-06T00:23:47.885 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-06T00:23:47.904 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-06T00:23:47.933 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3517-gb45ed440-1jammy [3,555 kB] 2024-05-06T00:23:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-06T00:23:47.943 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-06T00:23:47.956 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-06T00:23:47.960 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-06T00:23:47.961 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-06T00:23:47.961 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-06T00:23:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-06T00:23:47.973 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-06T00:23:47.975 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-06T00:23:47.977 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-06T00:23:47.981 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-06T00:23:47.987 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-06T00:23:47.991 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-06T00:23:47.993 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-06T00:23:47.995 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-06T00:23:47.997 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-06T00:23:47.998 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-06T00:23:48.001 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-06T00:23:48.005 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-06T00:23:48.005 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-06T00:23:48.006 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-06T00:23:48.011 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-06T00:23:48.013 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-06T00:23:48.013 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-06T00:23:48.013 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-06T00:23:48.013 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-06T00:23:48.014 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-06T00:23:48.017 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-06T00:23:48.019 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-06T00:23:48.020 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-06T00:23:48.026 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-06T00:23:48.029 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-06T00:23:48.031 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-06T00:23:48.032 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-06T00:23:48.033 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-06T00:23:48.033 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-06T00:23:48.035 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-06T00:23:48.036 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-06T00:23:48.046 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-06T00:23:48.047 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-06T00:23:48.047 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-06T00:23:48.049 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-06T00:23:48.049 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-06T00:23:48.049 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-06T00:23:48.049 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-06T00:23:48.053 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-06T00:23:48.053 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-06T00:23:48.054 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-06T00:23:48.055 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-06T00:23:48.055 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-06T00:23:48.066 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-06T00:23:48.066 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-06T00:23:48.067 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-06T00:23:48.067 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3517-gb45ed440-1jammy [3,223 kB] 2024-05-06T00:23:48.073 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-06T00:23:48.074 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-06T00:23:48.075 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-06T00:23:48.080 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-06T00:23:48.080 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-06T00:23:48.094 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-06T00:23:48.094 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-06T00:23:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-06T00:23:48.098 INFO:teuthology.orchestra.run.smithi078.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-05-06T00:23:48.099 INFO:teuthology.orchestra.run.smithi078.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-06T00:23:48.104 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-06T00:23:48.105 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-06T00:23:48.105 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-06T00:23:48.106 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-06T00:23:48.108 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-06T00:23:48.108 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-06T00:23:48.109 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-06T00:23:48.114 INFO:teuthology.orchestra.run.smithi078.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-06T00:23:48.118 INFO:teuthology.orchestra.run.smithi078.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-06T00:23:48.118 INFO:teuthology.orchestra.run.smithi078.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-06T00:23:48.119 INFO:teuthology.orchestra.run.smithi078.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-06T00:23:48.119 INFO:teuthology.orchestra.run.smithi078.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-06T00:23:48.123 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-06T00:23:48.125 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-06T00:23:48.126 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-06T00:23:48.135 INFO:teuthology.orchestra.run.smithi078.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-06T00:23:48.153 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-06T00:23:48.154 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-06T00:23:48.155 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-06T00:23:48.156 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-06T00:23:48.159 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-06T00:23:48.162 INFO:teuthology.orchestra.run.smithi186.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-05-06T00:23:48.164 INFO:teuthology.orchestra.run.smithi186.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-06T00:23:48.187 INFO:teuthology.orchestra.run.smithi186.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-06T00:23:48.188 INFO:teuthology.orchestra.run.smithi186.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-06T00:23:48.191 INFO:teuthology.orchestra.run.smithi078.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3517-gb45ed440-1jammy [712 kB] 2024-05-06T00:23:48.193 INFO:teuthology.orchestra.run.smithi186.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-06T00:23:48.196 INFO:teuthology.orchestra.run.smithi078.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-06T00:23:48.197 INFO:teuthology.orchestra.run.smithi078.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-06T00:23:48.198 INFO:teuthology.orchestra.run.smithi078.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-06T00:23:48.199 INFO:teuthology.orchestra.run.smithi078.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-06T00:23:48.200 INFO:teuthology.orchestra.run.smithi078.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-06T00:23:48.216 INFO:teuthology.orchestra.run.smithi186.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-06T00:23:48.218 INFO:teuthology.orchestra.run.smithi186.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-06T00:23:48.219 INFO:teuthology.orchestra.run.smithi078.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-06T00:23:48.221 INFO:teuthology.orchestra.run.smithi078.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3517-gb45ed440-1jammy [356 kB] 2024-05-06T00:23:48.221 INFO:teuthology.orchestra.run.smithi078.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-06T00:23:48.222 INFO:teuthology.orchestra.run.smithi078.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-06T00:23:48.223 INFO:teuthology.orchestra.run.smithi186.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-06T00:23:48.234 INFO:teuthology.orchestra.run.smithi078.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3517-gb45ed440-1jammy [32.8 kB] 2024-05-06T00:23:48.235 INFO:teuthology.orchestra.run.smithi078.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3517-gb45ed440-1jammy [180 kB] 2024-05-06T00:23:48.242 INFO:teuthology.orchestra.run.smithi078.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3517-gb45ed440-1jammy [67.4 kB] 2024-05-06T00:23:48.246 INFO:teuthology.orchestra.run.smithi078.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3517-gb45ed440-1jammy [334 kB] 2024-05-06T00:23:48.259 INFO:teuthology.orchestra.run.smithi078.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3517-gb45ed440-1jammy [6,892 kB] 2024-05-06T00:23:48.261 INFO:teuthology.orchestra.run.smithi078.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-06T00:23:48.262 INFO:teuthology.orchestra.run.smithi078.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-06T00:23:48.272 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-06T00:23:48.277 INFO:teuthology.orchestra.run.smithi078.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-06T00:23:48.278 INFO:teuthology.orchestra.run.smithi078.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-06T00:23:48.278 INFO:teuthology.orchestra.run.smithi078.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-06T00:23:48.279 INFO:teuthology.orchestra.run.smithi078.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-06T00:23:48.279 INFO:teuthology.orchestra.run.smithi078.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-06T00:23:48.279 INFO:teuthology.orchestra.run.smithi078.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-06T00:23:48.282 INFO:teuthology.orchestra.run.smithi078.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-06T00:23:48.285 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-06T00:23:48.300 INFO:teuthology.orchestra.run.smithi078.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-06T00:23:48.301 INFO:teuthology.orchestra.run.smithi078.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-06T00:23:48.302 INFO:teuthology.orchestra.run.smithi078.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-06T00:23:48.306 INFO:teuthology.orchestra.run.smithi186.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-06T00:23:48.307 INFO:teuthology.orchestra.run.smithi186.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-06T00:23:48.308 INFO:teuthology.orchestra.run.smithi186.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-06T00:23:48.322 INFO:teuthology.orchestra.run.smithi186.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-06T00:23:48.322 INFO:teuthology.orchestra.run.smithi186.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-06T00:23:48.322 INFO:teuthology.orchestra.run.smithi078.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-06T00:23:48.326 INFO:teuthology.orchestra.run.smithi078.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-06T00:23:48.327 INFO:teuthology.orchestra.run.smithi078.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-06T00:23:48.327 INFO:teuthology.orchestra.run.smithi078.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-06T00:23:48.332 INFO:teuthology.orchestra.run.smithi078.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-06T00:23:48.333 INFO:teuthology.orchestra.run.smithi078.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-06T00:23:48.335 INFO:teuthology.orchestra.run.smithi078.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-06T00:23:48.340 INFO:teuthology.orchestra.run.smithi186.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-06T00:23:48.342 INFO:teuthology.orchestra.run.smithi186.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-06T00:23:48.342 INFO:teuthology.orchestra.run.smithi186.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-06T00:23:48.347 INFO:teuthology.orchestra.run.smithi078.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-06T00:23:48.348 INFO:teuthology.orchestra.run.smithi078.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-06T00:23:48.349 INFO:teuthology.orchestra.run.smithi078.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-06T00:23:48.359 INFO:teuthology.orchestra.run.smithi186.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-06T00:23:48.359 INFO:teuthology.orchestra.run.smithi186.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-06T00:23:48.360 INFO:teuthology.orchestra.run.smithi186.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-06T00:23:48.360 INFO:teuthology.orchestra.run.smithi186.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-06T00:23:48.361 INFO:teuthology.orchestra.run.smithi186.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-06T00:23:48.361 INFO:teuthology.orchestra.run.smithi186.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-06T00:23:48.367 INFO:teuthology.orchestra.run.smithi078.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-06T00:23:48.376 INFO:teuthology.orchestra.run.smithi186.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-06T00:23:48.377 INFO:teuthology.orchestra.run.smithi186.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-06T00:23:48.380 INFO:teuthology.orchestra.run.smithi186.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-06T00:23:48.389 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-06T00:23:48.396 INFO:teuthology.orchestra.run.smithi186.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-06T00:23:48.397 INFO:teuthology.orchestra.run.smithi186.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-06T00:23:48.398 INFO:teuthology.orchestra.run.smithi186.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-06T00:23:48.402 INFO:teuthology.orchestra.run.smithi186.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-06T00:23:48.405 INFO:teuthology.orchestra.run.smithi186.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-06T00:23:48.405 INFO:teuthology.orchestra.run.smithi186.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-06T00:23:48.406 INFO:teuthology.orchestra.run.smithi186.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-06T00:23:48.410 INFO:teuthology.orchestra.run.smithi078.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-06T00:23:48.415 INFO:teuthology.orchestra.run.smithi186.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-06T00:23:48.416 INFO:teuthology.orchestra.run.smithi186.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-06T00:23:48.420 INFO:teuthology.orchestra.run.smithi186.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-06T00:23:48.435 INFO:teuthology.orchestra.run.smithi186.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-06T00:23:48.437 INFO:teuthology.orchestra.run.smithi186.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-06T00:23:48.437 INFO:teuthology.orchestra.run.smithi186.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-06T00:23:48.439 INFO:teuthology.orchestra.run.smithi186.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-06T00:23:48.440 INFO:teuthology.orchestra.run.smithi186.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-06T00:23:48.449 INFO:teuthology.orchestra.run.smithi186.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-06T00:23:48.449 INFO:teuthology.orchestra.run.smithi186.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-06T00:23:48.455 INFO:teuthology.orchestra.run.smithi186.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-06T00:23:48.458 INFO:teuthology.orchestra.run.smithi186.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-06T00:23:48.459 INFO:teuthology.orchestra.run.smithi186.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-06T00:23:48.549 INFO:teuthology.orchestra.run.smithi078.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3517-gb45ed440-1jammy [112 kB] 2024-05-06T00:23:48.553 INFO:teuthology.orchestra.run.smithi078.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3517-gb45ed440-1jammy [466 kB] 2024-05-06T00:23:48.562 INFO:teuthology.orchestra.run.smithi078.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-06T00:23:48.563 INFO:teuthology.orchestra.run.smithi078.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-06T00:23:48.564 INFO:teuthology.orchestra.run.smithi078.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-06T00:23:48.570 INFO:teuthology.orchestra.run.smithi078.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-06T00:23:48.572 INFO:teuthology.orchestra.run.smithi078.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-06T00:23:48.572 INFO:teuthology.orchestra.run.smithi078.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3517-gb45ed440-1jammy [25.8 MB] 2024-05-06T00:23:48.618 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-06T00:23:48.628 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-06T00:23:48.659 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-06T00:23:48.667 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-06T00:23:48.669 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-06T00:23:48.670 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-06T00:23:48.690 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-06T00:23:48.870 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3517-gb45ed440-1jammy [3,223 kB] 2024-05-06T00:23:48.870 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-06T00:23:49.308 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-06T00:23:49.347 INFO:teuthology.orchestra.run.smithi186.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3517-gb45ed440-1jammy [3,223 kB] 2024-05-06T00:23:49.348 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-06T00:23:49.350 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-06T00:23:49.480 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-06T00:23:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-06T00:23:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3517-gb45ed440-1jammy [712 kB] 2024-05-06T00:23:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-06T00:23:49.693 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-06T00:23:49.704 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-06T00:23:49.734 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-06T00:23:49.743 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-06T00:23:49.797 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3517-gb45ed440-1jammy [356 kB] 2024-05-06T00:23:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3517-gb45ed440-1jammy [5,095 kB] 2024-05-06T00:23:49.880 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3517-gb45ed440-1jammy [32.8 kB] 2024-05-06T00:23:49.881 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3517-gb45ed440-1jammy [180 kB] 2024-05-06T00:23:49.887 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3517-gb45ed440-1jammy [67.4 kB] 2024-05-06T00:23:49.916 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-06T00:23:49.930 INFO:teuthology.orchestra.run.smithi078.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3517-gb45ed440-1jammy [244 kB] 2024-05-06T00:23:49.939 INFO:teuthology.orchestra.run.smithi078.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3517-gb45ed440-1jammy [124 kB] 2024-05-06T00:23:49.941 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3517-gb45ed440-1jammy [334 kB] 2024-05-06T00:23:49.947 INFO:teuthology.orchestra.run.smithi078.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3517-gb45ed440-1jammy [1,544 kB] 2024-05-06T00:23:50.005 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-06T00:23:50.006 INFO:teuthology.orchestra.run.smithi186.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3517-gb45ed440-1jammy [712 kB] 2024-05-06T00:23:50.018 INFO:teuthology.orchestra.run.smithi078.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3517-gb45ed440-1jammy [6,259 kB] 2024-05-06T00:23:50.025 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3517-gb45ed440-1jammy [6,892 kB] 2024-05-06T00:23:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-06T00:23:50.052 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-06T00:23:50.096 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-06T00:23:50.145 INFO:teuthology.orchestra.run.smithi186.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3517-gb45ed440-1jammy [356 kB] 2024-05-06T00:23:50.188 INFO:teuthology.orchestra.run.smithi186.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3517-gb45ed440-1jammy [32.8 kB] 2024-05-06T00:23:50.222 INFO:teuthology.orchestra.run.smithi186.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3517-gb45ed440-1jammy [180 kB] 2024-05-06T00:23:50.249 INFO:teuthology.orchestra.run.smithi186.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3517-gb45ed440-1jammy [67.4 kB] 2024-05-06T00:23:50.254 INFO:teuthology.orchestra.run.smithi186.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3517-gb45ed440-1jammy [334 kB] 2024-05-06T00:23:50.309 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-06T00:23:50.316 INFO:teuthology.orchestra.run.smithi078.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3517-gb45ed440-1jammy [22.8 MB] 2024-05-06T00:23:50.321 INFO:teuthology.orchestra.run.smithi186.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3517-gb45ed440-1jammy [6,892 kB] 2024-05-06T00:23:50.322 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-06T00:23:50.330 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-06T00:23:50.334 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-06T00:23:50.356 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-06T00:23:50.359 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-06T00:23:50.526 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-06T00:23:50.860 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-06T00:23:50.893 INFO:teuthology.orchestra.run.smithi146.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-06T00:23:50.902 INFO:teuthology.orchestra.run.smithi146.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-06T00:23:51.105 INFO:teuthology.orchestra.run.smithi146.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-06T00:23:51.105 INFO:teuthology.orchestra.run.smithi146.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-06T00:23:51.112 INFO:teuthology.orchestra.run.smithi146.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-06T00:23:51.118 INFO:teuthology.orchestra.run.smithi146.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-06T00:23:51.132 INFO:teuthology.orchestra.run.smithi146.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-06T00:23:51.154 INFO:teuthology.orchestra.run.smithi146.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-05-06T00:23:51.160 INFO:teuthology.orchestra.run.smithi146.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-06T00:23:51.217 INFO:teuthology.orchestra.run.smithi186.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3517-gb45ed440-1jammy [112 kB] 2024-05-06T00:23:51.226 INFO:teuthology.orchestra.run.smithi186.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3517-gb45ed440-1jammy [466 kB] 2024-05-06T00:23:51.277 INFO:teuthology.orchestra.run.smithi186.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3517-gb45ed440-1jammy [25.8 MB] 2024-05-06T00:23:51.336 INFO:teuthology.orchestra.run.smithi146.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-06T00:23:51.598 INFO:teuthology.orchestra.run.smithi146.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-06T00:23:51.659 INFO:teuthology.orchestra.run.smithi078.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3517-gb45ed440-1jammy [14.1 kB] 2024-05-06T00:23:51.660 INFO:teuthology.orchestra.run.smithi078.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3517-gb45ed440-1jammy [870 kB] 2024-05-06T00:23:51.677 INFO:teuthology.orchestra.run.smithi078.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3517-gb45ed440-1jammy [2,438 kB] 2024-05-06T00:23:51.791 INFO:teuthology.orchestra.run.smithi146.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-06T00:23:51.814 INFO:teuthology.orchestra.run.smithi146.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-06T00:23:51.837 INFO:teuthology.orchestra.run.smithi078.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3517-gb45ed440-1jammy [791 kB] 2024-05-06T00:23:51.845 INFO:teuthology.orchestra.run.smithi146.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-06T00:23:51.851 INFO:teuthology.orchestra.run.smithi078.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3517-gb45ed440-1jammy [148 kB] 2024-05-06T00:23:51.855 INFO:teuthology.orchestra.run.smithi078.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3517-gb45ed440-1jammy [2,279 kB] 2024-05-06T00:23:51.962 INFO:teuthology.orchestra.run.smithi146.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3517-gb45ed440-1jammy [112 kB] 2024-05-06T00:23:51.975 INFO:teuthology.orchestra.run.smithi146.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3517-gb45ed440-1jammy [466 kB] 2024-05-06T00:23:51.976 INFO:teuthology.orchestra.run.smithi146.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-06T00:23:52.004 INFO:teuthology.orchestra.run.smithi078.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3517-gb45ed440-1jammy [8,625 kB] 2024-05-06T00:23:52.108 INFO:teuthology.orchestra.run.smithi146.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3517-gb45ed440-1jammy [25.8 MB] 2024-05-06T00:23:52.586 INFO:teuthology.orchestra.run.smithi078.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3517-gb45ed440-1jammy [14.2 kB] 2024-05-06T00:23:52.586 INFO:teuthology.orchestra.run.smithi078.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3517-gb45ed440-1jammy [51.7 MB] 2024-05-06T00:23:52.747 INFO:teuthology.orchestra.run.smithi146.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-06T00:23:52.749 INFO:teuthology.orchestra.run.smithi146.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-06T00:23:52.762 INFO:teuthology.orchestra.run.smithi146.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-06T00:23:52.762 INFO:teuthology.orchestra.run.smithi146.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-06T00:23:52.775 INFO:teuthology.orchestra.run.smithi146.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-06T00:23:53.094 INFO:teuthology.orchestra.run.smithi146.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-06T00:23:53.532 INFO:teuthology.orchestra.run.smithi146.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-06T00:23:53.536 INFO:teuthology.orchestra.run.smithi186.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3517-gb45ed440-1jammy [5,095 kB] 2024-05-06T00:23:53.537 INFO:teuthology.orchestra.run.smithi146.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-06T00:23:53.828 INFO:teuthology.orchestra.run.smithi146.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-06T00:23:53.829 INFO:teuthology.orchestra.run.smithi146.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-06T00:23:53.831 INFO:teuthology.orchestra.run.smithi146.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-06T00:23:53.832 INFO:teuthology.orchestra.run.smithi146.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-06T00:23:53.834 INFO:teuthology.orchestra.run.smithi146.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-06T00:23:53.836 INFO:teuthology.orchestra.run.smithi146.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-06T00:23:53.837 INFO:teuthology.orchestra.run.smithi146.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-06T00:23:53.839 INFO:teuthology.orchestra.run.smithi146.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-06T00:23:53.846 INFO:teuthology.orchestra.run.smithi186.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3517-gb45ed440-1jammy [244 kB] 2024-05-06T00:23:53.862 INFO:teuthology.orchestra.run.smithi186.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3517-gb45ed440-1jammy [124 kB] 2024-05-06T00:23:53.886 INFO:teuthology.orchestra.run.smithi186.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3517-gb45ed440-1jammy [1,544 kB] 2024-05-06T00:23:54.001 INFO:teuthology.orchestra.run.smithi186.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3517-gb45ed440-1jammy [6,259 kB] 2024-05-06T00:23:54.034 INFO:teuthology.orchestra.run.smithi146.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-06T00:23:54.493 INFO:teuthology.orchestra.run.smithi186.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3517-gb45ed440-1jammy [22.8 MB] 2024-05-06T00:23:54.504 INFO:teuthology.orchestra.run.smithi146.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-06T00:23:54.666 INFO:teuthology.orchestra.run.smithi146.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-06T00:23:54.717 INFO:teuthology.orchestra.run.smithi146.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-06T00:23:55.140 INFO:teuthology.orchestra.run.smithi146.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-06T00:23:55.511 INFO:teuthology.orchestra.run.smithi146.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-06T00:23:55.581 INFO:teuthology.orchestra.run.smithi146.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-06T00:23:55.609 INFO:teuthology.orchestra.run.smithi146.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-06T00:23:56.174 INFO:teuthology.orchestra.run.smithi146.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-06T00:23:56.243 INFO:teuthology.orchestra.run.smithi146.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-06T00:23:56.379 INFO:teuthology.orchestra.run.smithi186.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3517-gb45ed440-1jammy [14.1 kB] 2024-05-06T00:23:56.380 INFO:teuthology.orchestra.run.smithi186.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3517-gb45ed440-1jammy [870 kB] 2024-05-06T00:23:56.396 INFO:teuthology.orchestra.run.smithi186.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3517-gb45ed440-1jammy [2,438 kB] 2024-05-06T00:23:56.485 INFO:teuthology.orchestra.run.smithi078.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3517-gb45ed440-1jammy [124 kB] 2024-05-06T00:23:56.488 INFO:teuthology.orchestra.run.smithi078.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3517-gb45ed440-1jammy [38.9 kB] 2024-05-06T00:23:56.489 INFO:teuthology.orchestra.run.smithi078.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3517-gb45ed440-1jammy [13.6 MB] 2024-05-06T00:23:56.527 INFO:teuthology.orchestra.run.smithi186.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3517-gb45ed440-1jammy [791 kB] 2024-05-06T00:23:56.569 INFO:teuthology.orchestra.run.smithi186.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3517-gb45ed440-1jammy [148 kB] 2024-05-06T00:23:56.597 INFO:teuthology.orchestra.run.smithi186.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3517-gb45ed440-1jammy [2,279 kB] 2024-05-06T00:23:56.615 INFO:teuthology.orchestra.run.smithi146.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-06T00:23:56.765 INFO:teuthology.orchestra.run.smithi186.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3517-gb45ed440-1jammy [8,625 kB] 2024-05-06T00:23:56.784 INFO:teuthology.orchestra.run.smithi146.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-06T00:23:57.048 INFO:teuthology.orchestra.run.smithi146.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-06T00:23:57.066 INFO:teuthology.orchestra.run.smithi146.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-06T00:23:57.223 INFO:teuthology.orchestra.run.smithi146.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-06T00:23:57.295 INFO:teuthology.orchestra.run.smithi146.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-06T00:23:57.308 INFO:teuthology.orchestra.run.smithi078.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3517-gb45ed440-1jammy [91.4 kB] 2024-05-06T00:23:57.309 INFO:teuthology.orchestra.run.smithi078.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.0.0-3517-gb45ed440-1jammy [179 kB] 2024-05-06T00:23:57.355 INFO:teuthology.orchestra.run.smithi186.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3517-gb45ed440-1jammy [14.2 kB] 2024-05-06T00:23:57.356 INFO:teuthology.orchestra.run.smithi186.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3517-gb45ed440-1jammy [51.7 MB] 2024-05-06T00:23:57.876 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 181 MB in 10s (18.8 MB/s) 2024-05-06T00:23:57.979 INFO:teuthology.orchestra.run.smithi146.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-06T00:23:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-06T00:23:58.005 INFO:teuthology.orchestra.run.smithi146.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-06T00:23:58.034 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-06T00:23:58.132 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-05-06T00:23:58.138 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-06T00:23:58.152 INFO:teuthology.orchestra.run.smithi146.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-06T00:23:58.195 INFO:teuthology.orchestra.run.smithi146.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-06T00:23:58.208 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-06T00:23:58.337 INFO:teuthology.orchestra.run.smithi146.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3517-gb45ed440-1jammy [5,095 kB] 2024-05-06T00:23:58.578 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-06T00:23:58.604 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-06T00:23:58.644 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-06T00:23:58.989 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-06T00:23:59.015 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-06T00:23:59.037 INFO:teuthology.orchestra.run.smithi146.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3517-gb45ed440-1jammy [244 kB] 2024-05-06T00:23:59.056 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-06T00:23:59.081 INFO:teuthology.orchestra.run.smithi146.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3517-gb45ed440-1jammy [124 kB] 2024-05-06T00:23:59.108 INFO:teuthology.orchestra.run.smithi146.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3517-gb45ed440-1jammy [1,544 kB] 2024-05-06T00:23:59.326 INFO:teuthology.orchestra.run.smithi146.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3517-gb45ed440-1jammy [6,259 kB] 2024-05-06T00:23:59.434 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-06T00:23:59.460 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:23:59.509 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:23:59.744 INFO:teuthology.orchestra.run.smithi186.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3517-gb45ed440-1jammy [124 kB] 2024-05-06T00:23:59.746 INFO:teuthology.orchestra.run.smithi186.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3517-gb45ed440-1jammy [38.9 kB] 2024-05-06T00:23:59.747 INFO:teuthology.orchestra.run.smithi186.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3517-gb45ed440-1jammy [13.6 MB] 2024-05-06T00:23:59.929 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-06T00:23:59.955 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:23:59.995 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:24:00.134 INFO:teuthology.orchestra.run.smithi146.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3517-gb45ed440-1jammy [22.8 MB] 2024-05-06T00:24:00.195 INFO:teuthology.orchestra.run.smithi186.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3517-gb45ed440-1jammy [91.4 kB] 2024-05-06T00:24:00.272 INFO:teuthology.orchestra.run.smithi186.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.0.0-3517-gb45ed440-1jammy [179 kB] 2024-05-06T00:24:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-06T00:24:00.591 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:24:00.631 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:24:00.836 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 181 MB in 13s (14.4 MB/s) 2024-05-06T00:24:00.918 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-06T00:24:00.944 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-06T00:24:00.984 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-06T00:24:01.021 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-06T00:24:01.123 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-05-06T00:24:01.128 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-06T00:24:01.220 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-06T00:24:01.312 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librados2. 2024-05-06T00:24:01.339 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../007-librados2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:01.379 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librados2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:01.556 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-06T00:24:01.582 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-06T00:24:01.623 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-06T00:24:01.840 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librbd1. 2024-05-06T00:24:01.866 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../008-librbd1_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:01.906 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librbd1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:01.934 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-06T00:24:01.960 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-06T00:24:02.002 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-06T00:24:02.309 INFO:teuthology.orchestra.run.smithi146.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3517-gb45ed440-1jammy [14.1 kB] 2024-05-06T00:24:02.310 INFO:teuthology.orchestra.run.smithi146.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3517-gb45ed440-1jammy [870 kB] 2024-05-06T00:24:02.317 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs2. 2024-05-06T00:24:02.326 INFO:teuthology.orchestra.run.smithi146.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3517-gb45ed440-1jammy [2,438 kB] 2024-05-06T00:24:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:02.376 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:02.462 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-06T00:24:02.489 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:24:02.500 INFO:teuthology.orchestra.run.smithi146.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3517-gb45ed440-1jammy [791 kB] 2024-05-06T00:24:02.537 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:24:02.570 INFO:teuthology.orchestra.run.smithi146.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3517-gb45ed440-1jammy [148 kB] 2024-05-06T00:24:02.573 INFO:teuthology.orchestra.run.smithi146.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3517-gb45ed440-1jammy [2,279 kB] 2024-05-06T00:24:02.628 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rados. 2024-05-06T00:24:02.644 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:02.695 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rados (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:02.752 INFO:teuthology.orchestra.run.smithi146.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3517-gb45ed440-1jammy [8,625 kB] 2024-05-06T00:24:02.899 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-06T00:24:02.926 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:24:02.957 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:24:03.014 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-06T00:24:03.041 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:03.081 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-argparse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:03.277 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-06T00:24:03.304 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:24:03.342 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cephfs. 2024-05-06T00:24:03.343 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:24:03.357 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:03.360 INFO:teuthology.orchestra.run.smithi146.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3517-gb45ed440-1jammy [14.2 kB] 2024-05-06T00:24:03.400 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cephfs (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:03.400 INFO:teuthology.orchestra.run.smithi146.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3517-gb45ed440-1jammy [51.7 MB] 2024-05-06T00:24:03.694 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-06T00:24:03.697 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-06T00:24:03.720 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:03.723 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-06T00:24:03.753 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:03.763 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-06T00:24:04.047 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-06T00:24:04.058 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librados2. 2024-05-06T00:24:04.062 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-06T00:24:04.084 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../007-librados2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:04.097 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-06T00:24:04.124 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librados2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:04.408 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-prettytable. 2024-05-06T00:24:04.434 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-06T00:24:04.483 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-06T00:24:04.552 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librbd1. 2024-05-06T00:24:04.578 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../008-librbd1_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:04.618 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librbd1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:04.752 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rbd. 2024-05-06T00:24:04.767 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:04.810 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-06T00:24:05.189 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-06T00:24:05.230 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-06T00:24:05.499 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-06T00:24:05.514 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-06T00:24:05.557 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-06T00:24:05.664 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs2. 2024-05-06T00:24:05.690 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:05.750 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:05.871 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-06T00:24:05.897 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-06T00:24:05.937 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-06T00:24:06.016 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rados. 2024-05-06T00:24:06.042 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:06.074 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rados (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:06.274 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua5.1. 2024-05-06T00:24:06.300 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-06T00:24:06.341 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-06T00:24:06.402 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-06T00:24:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:06.469 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-argparse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:06.644 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-any. 2024-05-06T00:24:06.671 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-06T00:24:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-06T00:24:06.721 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cephfs. 2024-05-06T00:24:06.748 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:06.788 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cephfs (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:06.963 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package zip. 2024-05-06T00:24:06.978 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-06T00:24:07.021 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking zip (3.0-12build2) ... 2024-05-06T00:24:07.074 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-06T00:24:07.101 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:07.140 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:07.295 INFO:teuthology.orchestra.run.smithi146.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3517-gb45ed440-1jammy [124 kB] 2024-05-06T00:24:07.297 INFO:teuthology.orchestra.run.smithi146.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3517-gb45ed440-1jammy [38.9 kB] 2024-05-06T00:24:07.298 INFO:teuthology.orchestra.run.smithi146.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3517-gb45ed440-1jammy [13.6 MB] 2024-05-06T00:24:07.341 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package unzip. 2024-05-06T00:24:07.368 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-06T00:24:07.408 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-06T00:24:07.427 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-06T00:24:07.454 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-06T00:24:07.485 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-06T00:24:07.787 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package luarocks. 2024-05-06T00:24:07.813 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-prettytable. 2024-05-06T00:24:07.813 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-06T00:24:07.839 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-06T00:24:07.854 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-06T00:24:07.880 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-06T00:24:07.909 INFO:teuthology.orchestra.run.smithi146.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3517-gb45ed440-1jammy [91.4 kB] 2024-05-06T00:24:07.972 INFO:teuthology.orchestra.run.smithi146.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b45ed4400464709677363de54673bcce970bcb27/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.0.0-3517-gb45ed440-1jammy [179 kB] 2024-05-06T00:24:08.140 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rbd. 2024-05-06T00:24:08.167 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:08.207 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:08.228 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librgw2. 2024-05-06T00:24:08.255 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../025-librgw2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:08.295 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librgw2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:08.532 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 181 MB in 20s (8,929 kB/s) 2024-05-06T00:24:08.535 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-06T00:24:08.561 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-06T00:24:08.601 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-06T00:24:08.690 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-06T00:24:08.790 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-05-06T00:24:08.796 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-06T00:24:08.837 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-06T00:24:08.839 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rgw. 2024-05-06T00:24:08.852 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-06T00:24:08.866 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:08.880 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-06T00:24:08.887 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-06T00:24:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:09.174 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-06T00:24:09.200 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-06T00:24:09.208 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-06T00:24:09.234 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-06T00:24:09.240 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-06T00:24:09.250 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-06T00:24:09.275 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-06T00:24:09.361 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-06T00:24:09.361 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-06T00:24:09.544 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-06T00:24:09.559 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-06T00:24:09.568 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package lua5.1. 2024-05-06T00:24:09.594 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-06T00:24:09.594 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-06T00:24:09.612 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libradosstriper1. 2024-05-06T00:24:09.635 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-06T00:24:09.638 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libradosstriper1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:09.963 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package lua-any. 2024-05-06T00:24:09.978 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-06T00:24:10.014 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-06T00:24:10.021 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-06T00:24:10.027 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:24:10.070 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:24:10.107 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-common. 2024-05-06T00:24:10.134 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:10.173 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:10.257 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package zip. 2024-05-06T00:24:10.272 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-06T00:24:10.307 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking zip (3.0-12build2) ... 2024-05-06T00:24:10.602 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package unzip. 2024-05-06T00:24:10.628 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-06T00:24:10.650 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-06T00:24:10.668 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-06T00:24:10.676 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:24:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:24:11.053 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-06T00:24:11.080 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-06T00:24:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package luarocks. 2024-05-06T00:24:11.107 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-06T00:24:11.120 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:24:11.147 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-06T00:24:11.448 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-06T00:24:11.463 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-06T00:24:11.498 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-06T00:24:11.513 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librgw2. 2024-05-06T00:24:11.540 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../025-librgw2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:11.580 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librgw2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:11.654 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-base. 2024-05-06T00:24:11.681 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:11.730 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-base (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:11.784 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librados2. 2024-05-06T00:24:11.810 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../007-librados2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:11.850 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librados2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:12.124 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rgw. 2024-05-06T00:24:12.139 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:12.182 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:12.283 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-06T00:24:12.310 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-06T00:24:12.320 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librbd1. 2024-05-06T00:24:12.346 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../008-librbd1_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:12.350 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-06T00:24:12.386 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librbd1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:12.493 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-06T00:24:12.520 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-06T00:24:12.552 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-06T00:24:12.653 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cheroot. 2024-05-06T00:24:12.679 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-06T00:24:12.720 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-06T00:24:12.855 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libradosstriper1. 2024-05-06T00:24:12.856 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs2. 2024-05-06T00:24:12.881 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:12.882 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:12.921 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libradosstriper1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:12.922 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:13.040 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-06T00:24:13.066 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-06T00:24:13.106 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-06T00:24:13.183 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rados. 2024-05-06T00:24:13.198 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:13.233 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rados (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:13.349 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-common. 2024-05-06T00:24:13.376 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:13.384 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-06T00:24:13.410 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-06T00:24:13.416 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:13.442 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-06T00:24:13.544 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-06T00:24:13.571 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:13.611 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-argparse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-06T00:24:13.747 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-06T00:24:13.787 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-06T00:24:13.863 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cephfs. 2024-05-06T00:24:13.878 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:13.921 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cephfs (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:14.090 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tempora. 2024-05-06T00:24:14.116 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-06T00:24:14.156 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-06T00:24:14.199 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-06T00:24:14.214 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:14.266 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:14.459 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-portend. 2024-05-06T00:24:14.486 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-06T00:24:14.526 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-06T00:24:14.562 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-06T00:24:14.589 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-06T00:24:14.629 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-06T00:24:14.763 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-base. 2024-05-06T00:24:14.789 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:14.796 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-06T00:24:14.822 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-06T00:24:14.831 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-base (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:14.854 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-06T00:24:14.956 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-prettytable. 2024-05-06T00:24:14.983 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-06T00:24:15.023 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-06T00:24:15.165 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-06T00:24:15.192 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-06T00:24:15.232 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-06T00:24:15.284 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rbd. 2024-05-06T00:24:15.299 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:15.334 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:15.401 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-06T00:24:15.427 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-06T00:24:15.467 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-06T00:24:15.586 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-natsort. 2024-05-06T00:24:15.612 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-06T00:24:15.652 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-06T00:24:15.662 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-06T00:24:15.688 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-06T00:24:15.728 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-06T00:24:15.945 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cheroot. 2024-05-06T00:24:15.971 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-06T00:24:15.981 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-logutils. 2024-05-06T00:24:16.008 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-05-06T00:24:16.011 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-06T00:24:16.047 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-06T00:24:16.139 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-06T00:24:16.165 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-06T00:24:16.206 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-06T00:24:16.290 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-06T00:24:16.317 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-06T00:24:16.325 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-06T00:24:16.352 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-06T00:24:16.356 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-06T00:24:16.392 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-06T00:24:16.517 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-06T00:24:16.543 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-06T00:24:16.583 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-06T00:24:16.642 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-06T00:24:16.669 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-06T00:24:16.670 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-mako. 2024-05-06T00:24:16.685 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-06T00:24:16.709 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-06T00:24:16.728 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-06T00:24:16.903 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua5.1. 2024-05-06T00:24:16.918 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-06T00:24:16.961 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-06T00:24:17.014 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-06T00:24:17.038 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-06T00:24:17.048 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-06T00:24:17.075 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-06T00:24:17.086 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-06T00:24:17.115 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-06T00:24:17.281 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-any. 2024-05-06T00:24:17.308 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-06T00:24:17.348 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-tempora. 2024-05-06T00:24:17.348 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-06T00:24:17.364 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-06T00:24:17.398 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-06T00:24:17.418 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-06T00:24:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-06T00:24:17.485 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-06T00:24:17.617 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package zip. 2024-05-06T00:24:17.643 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-06T00:24:17.683 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking zip (3.0-12build2) ... 2024-05-06T00:24:17.709 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-portend. 2024-05-06T00:24:17.735 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-06T00:24:17.745 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webob. 2024-05-06T00:24:17.761 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-06T00:24:17.775 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-06T00:24:17.795 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-06T00:24:18.011 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package unzip. 2024-05-06T00:24:18.038 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-06T00:24:18.078 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-06T00:24:18.103 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-06T00:24:18.105 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-06T00:24:18.145 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-06T00:24:18.157 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-waitress. 2024-05-06T00:24:18.183 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-06T00:24:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-06T00:24:18.431 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-06T00:24:18.440 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package luarocks. 2024-05-06T00:24:18.447 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-06T00:24:18.455 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-06T00:24:18.481 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-06T00:24:18.490 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-06T00:24:18.552 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tempita. 2024-05-06T00:24:18.578 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-06T00:24:18.643 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-06T00:24:18.826 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-natsort. 2024-05-06T00:24:18.853 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-06T00:24:18.865 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librgw2. 2024-05-06T00:24:18.891 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../025-librgw2_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:18.892 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-06T00:24:18.931 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librgw2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:18.955 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-paste. 2024-05-06T00:24:18.981 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-06T00:24:19.021 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-06T00:24:19.204 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-logutils. 2024-05-06T00:24:19.231 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-05-06T00:24:19.271 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-06T00:24:19.366 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-06T00:24:19.381 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-06T00:24:19.424 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-06T00:24:19.483 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rgw. 2024-05-06T00:24:19.499 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:19.532 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-06T00:24:19.541 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:19.558 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-06T00:24:19.590 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-06T00:24:19.727 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-06T00:24:19.754 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-06T00:24:19.794 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-06T00:24:19.869 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-06T00:24:19.876 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-mako. 2024-05-06T00:24:19.884 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-06T00:24:19.903 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-06T00:24:19.919 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-06T00:24:19.943 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-06T00:24:20.089 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webtest. 2024-05-06T00:24:20.115 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-05-06T00:24:20.147 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-06T00:24:20.408 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pecan. 2024-05-06T00:24:20.435 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-06T00:24:20.475 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-06T00:24:20.830 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-certifi. 2024-05-06T00:24:20.857 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-05-06T00:24:20.897 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-06T00:24:20.964 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-06T00:24:20.991 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-06T00:24:21.030 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-06T00:24:21.064 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libradosstriper1. 2024-05-06T00:24:21.092 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:21.131 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libradosstriper1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:21.333 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-06T00:24:21.360 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-06T00:24:21.399 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-06T00:24:21.493 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-common. 2024-05-06T00:24:21.519 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:21.652 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webob. 2024-05-06T00:24:21.679 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-06T00:24:21.710 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-06T00:24:22.047 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-waitress. 2024-05-06T00:24:22.059 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-idna. 2024-05-06T00:24:22.073 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-06T00:24:22.086 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-05-06T00:24:22.117 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-idna (3.3-1) ... 2024-05-06T00:24:22.117 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-06T00:24:22.416 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-tempita. 2024-05-06T00:24:22.428 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-urllib3. 2024-05-06T00:24:22.443 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-06T00:24:22.455 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-06T00:24:22.474 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-06T00:24:22.495 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-06T00:24:22.761 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-paste. 2024-05-06T00:24:22.787 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-06T00:24:22.807 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-requests. 2024-05-06T00:24:22.827 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-06T00:24:22.833 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-06T00:24:22.873 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-06T00:24:23.014 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-base. 2024-05-06T00:24:23.041 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:23.090 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-base (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:23.157 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-06T00:24:23.182 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-06T00:24:23.184 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-06T00:24:23.211 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-06T00:24:23.222 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-06T00:24:23.242 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-06T00:24:23.508 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-06T00:24:23.521 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-06T00:24:23.535 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-06T00:24:23.548 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:23.575 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-06T00:24:23.587 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:23.609 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-06T00:24:23.625 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-06T00:24:23.659 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-06T00:24:23.844 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webtest. 2024-05-06T00:24:23.871 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-05-06T00:24:23.902 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-06T00:24:23.946 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cheroot. 2024-05-06T00:24:23.972 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-06T00:24:23.977 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-06T00:24:24.002 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:24.012 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-06T00:24:24.042 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:24.206 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pecan. 2024-05-06T00:24:24.232 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-06T00:24:24.297 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-06T00:24:24.362 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-06T00:24:24.365 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-06T00:24:24.388 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-06T00:24:24.392 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-06T00:24:24.429 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-06T00:24:24.431 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-06T00:24:24.677 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-certifi. 2024-05-06T00:24:24.703 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-05-06T00:24:24.734 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-06T00:24:24.743 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-06T00:24:24.761 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-06T00:24:24.773 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-openssl. 2024-05-06T00:24:24.800 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-05-06T00:24:24.800 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-06T00:24:24.848 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-06T00:24:25.062 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-06T00:24:25.080 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-idna. 2024-05-06T00:24:25.089 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-06T00:24:25.107 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-05-06T00:24:25.128 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-06T00:24:25.146 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-idna (3.3-1) ... 2024-05-06T00:24:25.251 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr. 2024-05-06T00:24:25.278 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:25.326 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:25.399 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-urllib3. 2024-05-06T00:24:25.414 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-06T00:24:25.415 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempora. 2024-05-06T00:24:25.441 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-06T00:24:25.449 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-06T00:24:25.481 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-06T00:24:25.711 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-requests. 2024-05-06T00:24:25.730 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mon. 2024-05-06T00:24:25.737 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-06T00:24:25.756 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:25.767 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-portend. 2024-05-06T00:24:25.777 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-06T00:24:25.794 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-06T00:24:25.796 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mon (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:25.833 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-06T00:24:26.088 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-06T00:24:26.111 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-06T00:24:26.115 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-06T00:24:26.138 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-06T00:24:26.155 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-06T00:24:26.178 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-06T00:24:26.441 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-06T00:24:26.457 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:26.458 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-osd. 2024-05-06T00:24:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:26.499 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:26.533 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-osd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:26.623 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-06T00:24:26.649 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-06T00:24:26.689 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-06T00:24:26.862 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-06T00:24:26.890 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:26.929 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:27.001 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-natsort. 2024-05-06T00:24:27.028 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-06T00:24:27.067 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-06T00:24:27.215 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-06T00:24:27.242 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-06T00:24:27.282 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-06T00:24:27.378 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-logutils. 2024-05-06T00:24:27.405 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-05-06T00:24:27.445 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-06T00:24:27.618 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-openssl. 2024-05-06T00:24:27.645 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-05-06T00:24:27.670 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph. 2024-05-06T00:24:27.684 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-06T00:24:27.696 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../066-ceph_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:27.723 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-06T00:24:27.736 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:27.738 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-06T00:24:27.773 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-06T00:24:28.029 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr. 2024-05-06T00:24:28.056 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:28.059 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-mako. 2024-05-06T00:24:28.081 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-fuse. 2024-05-06T00:24:28.085 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-06T00:24:28.104 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:28.107 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:28.125 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-06T00:24:28.147 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:28.449 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mon. 2024-05-06T00:24:28.461 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-06T00:24:28.476 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:28.488 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-06T00:24:28.507 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mon (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:28.528 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-06T00:24:28.559 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mds. 2024-05-06T00:24:28.586 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:28.634 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mds (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:28.831 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-06T00:24:28.858 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-06T00:24:28.898 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-06T00:24:29.111 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-osd. 2024-05-06T00:24:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package cephadm. 2024-05-06T00:24:29.137 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:29.142 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webob. 2024-05-06T00:24:29.147 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../069-cephadm_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:29.169 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-06T00:24:29.177 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-osd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:29.195 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:29.200 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-06T00:24:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-06T00:24:29.528 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-waitress. 2024-05-06T00:24:29.533 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-06T00:24:29.555 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-06T00:24:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-06T00:24:29.598 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-06T00:24:29.906 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempita. 2024-05-06T00:24:29.926 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-06T00:24:29.933 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-06T00:24:29.941 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-06T00:24:29.972 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-06T00:24:29.984 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-06T00:24:30.197 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph. 2024-05-06T00:24:30.224 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../066-ceph_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:30.264 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:30.275 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-paste. 2024-05-06T00:24:30.302 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-06T00:24:30.341 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-06T00:24:30.583 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-fuse. 2024-05-06T00:24:30.610 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:30.644 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-06T00:24:30.649 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:30.660 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-06T00:24:30.703 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-06T00:24:31.023 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-babel. 2024-05-06T00:24:31.028 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mds. 2024-05-06T00:24:31.039 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-06T00:24:31.051 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-06T00:24:31.054 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:31.066 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-06T00:24:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-06T00:24:31.105 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-06T00:24:31.111 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mds (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:31.409 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webtest. 2024-05-06T00:24:31.417 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jinja2. 2024-05-06T00:24:31.435 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-05-06T00:24:31.444 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-06T00:24:31.475 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-06T00:24:31.484 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-06T00:24:31.647 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package cephadm. 2024-05-06T00:24:31.674 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../069-cephadm_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:31.714 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:31.786 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pecan. 2024-05-06T00:24:31.813 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-06T00:24:31.820 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-06T00:24:31.844 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-06T00:24:31.847 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:31.887 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:32.025 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-06T00:24:32.052 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-06T00:24:32.091 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-06T00:24:32.199 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-certifi. 2024-05-06T00:24:32.226 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-05-06T00:24:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-06T00:24:32.265 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-06T00:24:32.276 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-05-06T00:24:32.315 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-06T00:24:32.394 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-06T00:24:32.421 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-06T00:24:32.460 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-06T00:24:32.568 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-idna. 2024-05-06T00:24:32.595 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-05-06T00:24:32.627 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-routes. 2024-05-06T00:24:32.635 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-idna (3.3-1) ... 2024-05-06T00:24:32.653 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-06T00:24:32.693 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-06T00:24:32.954 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-urllib3. 2024-05-06T00:24:32.981 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-06T00:24:33.013 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-06T00:24:33.029 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-06T00:24:33.041 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:33.080 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:33.349 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-requests. 2024-05-06T00:24:33.376 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-06T00:24:33.415 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-06T00:24:33.468 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-babel. 2024-05-06T00:24:33.495 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-06T00:24:33.534 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-06T00:24:33.726 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-06T00:24:33.753 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-06T00:24:33.793 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-06T00:24:33.862 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jinja2. 2024-05-06T00:24:33.889 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-06T00:24:33.928 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-06T00:24:34.088 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-06T00:24:34.115 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:34.154 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:34.273 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-06T00:24:34.300 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:34.318 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-06T00:24:34.331 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:34.345 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-06T00:24:34.385 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-06T00:24:34.609 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-06T00:24:34.625 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:34.667 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:34.668 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-06T00:24:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-05-06T00:24:34.734 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-06T00:24:34.914 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-joblib. 2024-05-06T00:24:34.941 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-06T00:24:34.978 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-routes. 2024-05-06T00:24:34.980 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-06T00:24:34.986 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-06T00:24:34.994 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-06T00:24:35.013 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-06T00:24:35.030 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-06T00:24:35.053 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-06T00:24:35.275 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-06T00:24:35.290 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-06T00:24:35.333 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-06T00:24:35.340 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-06T00:24:35.356 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-openssl. 2024-05-06T00:24:35.367 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:35.382 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-05-06T00:24:35.407 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:35.422 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-06T00:24:35.670 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn. 2024-05-06T00:24:35.696 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-06T00:24:35.734 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr. 2024-05-06T00:24:35.736 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-06T00:24:35.761 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:35.800 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:36.187 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mon. 2024-05-06T00:24:36.214 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:36.253 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mon (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-06T00:24:36.448 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:36.581 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:36.653 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-06T00:24:36.669 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-06T00:24:36.712 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-06T00:24:37.049 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-osd. 2024-05-06T00:24:37.075 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:37.115 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-osd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:37.240 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-joblib. 2024-05-06T00:24:37.268 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-06T00:24:37.307 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-06T00:24:37.476 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cachetools. 2024-05-06T00:24:37.504 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-05-06T00:24:37.543 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-06T00:24:37.702 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-06T00:24:37.730 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-06T00:24:37.768 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-06T00:24:38.063 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-06T00:24:38.079 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn. 2024-05-06T00:24:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-06T00:24:38.107 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-06T00:24:38.129 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-06T00:24:38.146 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-06T00:24:38.151 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph. 2024-05-06T00:24:38.179 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../066-ceph_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:38.218 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:38.466 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-06T00:24:38.478 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-fuse. 2024-05-06T00:24:38.493 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-06T00:24:38.494 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:38.532 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-06T00:24:38.536 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:38.819 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rsa. 2024-05-06T00:24:38.821 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-06T00:24:38.834 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-05-06T00:24:38.848 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:38.869 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-06T00:24:38.915 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mds. 2024-05-06T00:24:38.942 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:38.973 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:38.981 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mds (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:39.173 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-google-auth. 2024-05-06T00:24:39.200 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-05-06T00:24:39.239 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-06T00:24:39.393 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package cephadm. 2024-05-06T00:24:39.419 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../069-cephadm_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:39.459 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:39.584 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-06T00:24:39.611 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-06T00:24:39.650 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-06T00:24:39.753 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-06T00:24:39.780 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-06T00:24:39.820 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-06T00:24:39.868 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cachetools. 2024-05-06T00:24:39.895 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-05-06T00:24:39.934 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-06T00:24:39.962 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-websocket. 2024-05-06T00:24:39.989 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-05-06T00:24:40.028 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-06T00:24:40.148 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-06T00:24:40.174 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-06T00:24:40.214 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-06T00:24:40.237 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-06T00:24:40.265 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-06T00:24:40.304 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-06T00:24:40.324 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-06T00:24:40.353 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-06T00:24:40.390 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-06T00:24:40.641 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-06T00:24:40.668 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-06T00:24:40.707 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-06T00:24:41.010 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rsa. 2024-05-06T00:24:41.021 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-06T00:24:41.038 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-05-06T00:24:41.038 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:41.068 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-06T00:24:41.079 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:41.252 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-babel. 2024-05-06T00:24:41.280 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-06T00:24:41.319 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-06T00:24:41.338 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-google-auth. 2024-05-06T00:24:41.362 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-05-06T00:24:41.448 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-06T00:24:41.549 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-06T00:24:41.577 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-06T00:24:41.607 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-06T00:24:41.800 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-06T00:24:41.827 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-06T00:24:41.847 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jinja2. 2024-05-06T00:24:41.866 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-06T00:24:41.873 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-06T00:24:41.913 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-06T00:24:41.936 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-06T00:24:41.963 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-06T00:24:42.002 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-06T00:24:42.178 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-websocket. 2024-05-06T00:24:42.205 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-05-06T00:24:42.241 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-06T00:24:42.244 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-06T00:24:42.254 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package jq. 2024-05-06T00:24:42.268 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:42.271 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-06T00:24:42.307 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:42.313 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-06T00:24:42.556 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-06T00:24:42.566 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package socat. 2024-05-06T00:24:42.581 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-06T00:24:42.583 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-06T00:24:42.622 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-06T00:24:42.624 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-06T00:24:42.627 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-06T00:24:42.654 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-05-06T00:24:42.694 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-06T00:24:42.911 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package xmlstarlet. 2024-05-06T00:24:42.927 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-06T00:24:42.960 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-06T00:24:42.963 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-routes. 2024-05-06T00:24:42.990 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-06T00:24:43.030 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-06T00:24:43.294 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-06T00:24:43.310 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:43.344 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:43.359 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-test. 2024-05-06T00:24:43.366 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-06T00:24:43.386 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:43.393 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:43.427 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-test (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:43.433 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:43.648 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-06T00:24:43.674 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-06T00:24:43.713 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-06T00:24:44.059 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-06T00:24:44.086 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-06T00:24:44.125 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-06T00:24:44.411 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package jq. 2024-05-06T00:24:44.438 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-06T00:24:44.477 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-06T00:24:44.688 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package socat. 2024-05-06T00:24:44.696 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-06T00:24:44.704 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-06T00:24:44.712 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-06T00:24:44.746 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-06T00:24:44.754 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-06T00:24:45.017 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package xmlstarlet. 2024-05-06T00:24:45.032 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-06T00:24:45.066 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-06T00:24:45.241 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-joblib. 2024-05-06T00:24:45.268 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-06T00:24:45.299 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-06T00:24:45.395 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-test. 2024-05-06T00:24:45.423 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:45.462 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-test (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:45.646 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-volume. 2024-05-06T00:24:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-06T00:24:45.674 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:45.682 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-06T00:24:45.704 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-volume (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:45.719 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-06T00:24:46.017 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-06T00:24:46.022 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn. 2024-05-06T00:24:46.033 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:46.049 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-06T00:24:46.067 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs-dev (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:46.091 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-06T00:24:46.320 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-06T00:24:46.348 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-06T00:24:46.386 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-06T00:24:46.725 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-06T00:24:46.752 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-05-06T00:24:46.791 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-06T00:24:46.846 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-06T00:24:46.873 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:47.015 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:47.162 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package nvme-cli. 2024-05-06T00:24:47.190 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-06T00:24:47.228 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-06T00:24:47.634 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package pkg-config. 2024-05-06T00:24:47.661 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-06T00:24:47.720 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-06T00:24:47.775 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-volume. 2024-05-06T00:24:47.802 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:47.833 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-volume (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:47.978 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-06T00:24:47.994 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-06T00:24:48.045 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-06T00:24:48.120 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-06T00:24:48.138 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:48.160 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cachetools. 2024-05-06T00:24:48.169 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs-dev (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:48.188 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-05-06T00:24:48.226 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-06T00:24:48.456 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-06T00:24:48.481 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-06T00:24:48.483 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-06T00:24:48.509 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-06T00:24:48.522 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-06T00:24:48.529 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-06T00:24:48.548 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-06T00:24:48.556 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-06T00:24:48.595 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-06T00:24:48.844 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-06T00:24:48.871 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-05-06T00:24:48.910 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-06T00:24:48.918 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastescript. 2024-05-06T00:24:48.932 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-06T00:24:48.945 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-05-06T00:24:48.959 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-06T00:24:48.992 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-06T00:24:48.998 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-06T00:24:49.297 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package nvme-cli. 2024-05-06T00:24:49.310 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rsa. 2024-05-06T00:24:49.324 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-06T00:24:49.337 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-05-06T00:24:49.338 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pluggy. 2024-05-06T00:24:49.366 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-06T00:24:49.366 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-06T00:24:49.376 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-06T00:24:49.404 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-06T00:24:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-google-auth. 2024-05-06T00:24:49.699 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-psutil. 2024-05-06T00:24:49.700 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-05-06T00:24:49.726 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-06T00:24:49.729 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-06T00:24:49.735 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package pkg-config. 2024-05-06T00:24:49.763 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-06T00:24:49.774 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-06T00:24:49.794 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-06T00:24:49.991 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-06T00:24:50.007 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-06T00:24:50.021 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-06T00:24:50.037 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-06T00:24:50.049 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-06T00:24:50.080 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-06T00:24:50.119 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-py. 2024-05-06T00:24:50.146 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-05-06T00:24:50.193 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-06T00:24:50.361 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-websocket. 2024-05-06T00:24:50.387 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-05-06T00:24:50.427 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-06T00:24:50.466 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-06T00:24:50.494 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-06T00:24:50.496 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pygments. 2024-05-06T00:24:50.524 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-06T00:24:50.525 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-06T00:24:50.563 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-06T00:24:50.747 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-06T00:24:50.774 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-06T00:24:50.813 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-06T00:24:50.827 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastescript. 2024-05-06T00:24:50.855 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-05-06T00:24:50.910 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-06T00:24:51.026 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-06T00:24:51.054 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-06T00:24:51.092 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-06T00:24:51.230 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pluggy. 2024-05-06T00:24:51.258 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-06T00:24:51.296 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-06T00:24:51.387 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-toml. 2024-05-06T00:24:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-05-06T00:24:51.453 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-06T00:24:51.460 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-06T00:24:51.477 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:51.518 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:51.616 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-psutil. 2024-05-06T00:24:51.644 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-06T00:24:51.683 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-06T00:24:51.756 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pytest. 2024-05-06T00:24:51.784 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-06T00:24:51.788 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-06T00:24:51.817 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-06T00:24:51.846 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-06T00:24:51.847 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-06T00:24:52.019 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-py. 2024-05-06T00:24:52.047 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-05-06T00:24:52.094 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-06T00:24:52.185 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplejson. 2024-05-06T00:24:52.191 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-06T00:24:52.213 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-06T00:24:52.218 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-06T00:24:52.251 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-06T00:24:52.257 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-06T00:24:52.439 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pygments. 2024-05-06T00:24:52.466 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-06T00:24:52.505 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-06T00:24:52.520 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-06T00:24:52.526 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package jq. 2024-05-06T00:24:52.537 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-06T00:24:52.542 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-06T00:24:52.579 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-06T00:24:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-06T00:24:52.829 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package socat. 2024-05-06T00:24:52.845 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-06T00:24:52.879 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-06T00:24:52.943 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-06T00:24:52.970 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-06T00:24:53.001 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-06T00:24:53.176 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package xmlstarlet. 2024-05-06T00:24:53.203 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-06T00:24:53.242 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-06T00:24:53.287 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-toml. 2024-05-06T00:24:53.300 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package radosgw. 2024-05-06T00:24:53.314 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-05-06T00:24:53.328 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../116-radosgw_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:53.353 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-06T00:24:53.579 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-test. 2024-05-06T00:24:53.583 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking radosgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:53.595 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:53.629 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-test (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:53.658 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pytest. 2024-05-06T00:24:53.684 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-06T00:24:53.722 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-06T00:24:54.026 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplejson. 2024-05-06T00:24:54.054 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-06T00:24:54.092 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-06T00:24:54.370 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-06T00:24:54.386 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-06T00:24:54.394 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package rbd-fuse. 2024-05-06T00:24:54.410 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:54.428 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-06T00:24:54.453 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking rbd-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:54.680 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package rbd-nbd. 2024-05-06T00:24:54.696 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../118-rbd-nbd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:54.730 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking rbd-nbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:55.043 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-06T00:24:55.182 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package radosgw. 2024-05-06T00:24:55.210 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../116-radosgw_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:55.257 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking radosgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:55.302 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-06T00:24:55.428 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-06T00:24:55.696 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-06T00:24:55.850 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-volume. 2024-05-06T00:24:55.878 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3517-gb45ed440-1jammy_all.deb ... 2024-05-06T00:24:55.917 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-volume (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:56.035 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package rbd-fuse. 2024-05-06T00:24:56.042 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-06T00:24:56.048 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:56.093 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking rbd-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:56.212 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-06T00:24:56.228 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:56.270 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs-dev (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:56.354 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package rbd-nbd. 2024-05-06T00:24:56.370 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../118-rbd-nbd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:24:56.412 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking rbd-nbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:56.458 INFO:teuthology.orchestra.run.smithi078.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-06T00:24:56.522 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:56.573 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-06T00:24:56.601 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-06T00:24:56.664 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-06T00:24:56.725 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-06T00:24:56.975 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-06T00:24:57.002 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-06T00:24:57.029 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user cephadm....done 2024-05-06T00:24:57.030 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-05-06T00:24:57.069 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-06T00:24:57.077 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-06T00:24:57.178 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-06T00:24:57.312 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-06T00:24:57.381 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package nvme-cli. 2024-05-06T00:24:57.409 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-06T00:24:57.447 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-06T00:24:57.454 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-06T00:24:57.627 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-06T00:24:57.714 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-06T00:24:57.840 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-06T00:24:58.061 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package pkg-config. 2024-05-06T00:24:58.076 INFO:teuthology.orchestra.run.smithi186.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-06T00:24:58.089 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-06T00:24:58.100 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-06T00:24:58.127 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-06T00:24:58.145 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:24:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-06T00:24:58.389 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-06T00:24:58.417 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-06T00:24:58.455 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-06T00:24:58.469 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-06T00:24:58.702 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user cephadm....done 2024-05-06T00:24:58.771 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-06T00:24:58.868 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-06T00:24:58.875 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-06T00:24:58.903 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-06T00:24:58.941 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-06T00:24:59.114 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-06T00:24:59.144 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-06T00:24:59.244 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastescript. 2024-05-06T00:24:59.272 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-05-06T00:24:59.311 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-06T00:24:59.374 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-06T00:24:59.395 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-06T00:24:59.521 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-06T00:24:59.631 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pluggy. 2024-05-06T00:24:59.634 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-06T00:24:59.659 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-06T00:24:59.698 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-06T00:24:59.790 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-06T00:24:59.860 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-06T00:24:59.992 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-psutil. 2024-05-06T00:25:00.019 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-06T00:25:00.048 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-06T00:25:00.058 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-06T00:25:00.120 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-06T00:25:00.166 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-06T00:25:00.362 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-06T00:25:00.394 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-py. 2024-05-06T00:25:00.422 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-05-06T00:25:00.460 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-06T00:25:00.469 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-06T00:25:00.597 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-argparse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:00.789 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pygments. 2024-05-06T00:25:00.817 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-06T00:25:00.819 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-06T00:25:00.855 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-06T00:25:00.923 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-06T00:25:01.049 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-06T00:25:01.129 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-06T00:25:01.175 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-06T00:25:01.301 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-06T00:25:01.359 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-06T00:25:01.376 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-06T00:25:01.414 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-06T00:25:01.418 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-06T00:25:01.569 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-06T00:25:01.657 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-06T00:25:01.671 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-toml. 2024-05-06T00:25:01.698 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-05-06T00:25:01.737 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-06T00:25:01.916 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-06T00:25:01.971 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-06T00:25:02.001 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-06T00:25:02.023 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pytest. 2024-05-06T00:25:02.051 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-06T00:25:02.064 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-06T00:25:02.089 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-06T00:25:02.151 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-06T00:25:02.190 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-06T00:25:02.410 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-argparse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:02.418 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplejson. 2024-05-06T00:25:02.446 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-06T00:25:02.484 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-06T00:25:02.558 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-06T00:25:02.685 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-06T00:25:02.770 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-06T00:25:02.770 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-06T00:25:02.786 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-06T00:25:02.828 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-06T00:25:02.917 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-06T00:25:02.961 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-06T00:25:03.047 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-06T00:25:03.173 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-06T00:25:03.237 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-06T00:25:03.432 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-06T00:25:03.538 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up zip (3.0-12build2) ... 2024-05-06T00:25:03.549 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package radosgw. 2024-05-06T00:25:03.577 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../116-radosgw_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:25:03.616 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking radosgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:03.639 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-06T00:25:03.807 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-06T00:25:03.839 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-06T00:25:03.894 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-06T00:25:03.994 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-06T00:25:04.358 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-06T00:25:04.370 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-06T00:25:04.452 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package rbd-fuse. 2024-05-06T00:25:04.480 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:25:04.497 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-06T00:25:04.518 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking rbd-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:04.609 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-06T00:25:04.735 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-06T00:25:04.763 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package rbd-nbd. 2024-05-06T00:25:04.764 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-06T00:25:04.791 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../118-rbd-nbd_19.0.0-3517-gb45ed440-1jammy_amd64.deb ... 2024-05-06T00:25:04.821 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking rbd-nbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:04.861 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-06T00:25:05.032 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-06T00:25:05.117 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-06T00:25:05.154 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-06T00:25:05.334 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up zip (3.0-12build2) ... 2024-05-06T00:25:05.392 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-06T00:25:05.434 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-06T00:25:05.493 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-06T00:25:05.522 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-06T00:25:05.747 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-06T00:25:05.782 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-06T00:25:06.082 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-06T00:25:06.086 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-06T00:25:06.166 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-06T00:25:06.329 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-06T00:25:06.443 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-06T00:25:06.455 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-06T00:25:06.530 INFO:teuthology.orchestra.run.smithi146.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-06T00:25:06.581 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-06T00:25:06.629 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:06.777 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-06T00:25:06.882 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-06T00:25:07.028 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-06T00:25:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:07.283 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-06T00:25:07.406 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-idna (3.3-1) ... 2024-05-06T00:25:07.488 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user cephadm....done 2024-05-06T00:25:07.551 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-06T00:25:07.636 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-06T00:25:07.715 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-06T00:25:07.887 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-06T00:25:07.936 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-06T00:25:08.138 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-06T00:25:08.254 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-06T00:25:08.264 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-06T00:25:08.532 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-06T00:25:08.589 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-06T00:25:08.791 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-06T00:25:08.813 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-06T00:25:08.840 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-06T00:25:08.917 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-06T00:25:08.966 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:09.065 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:09.191 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-06T00:25:09.202 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-06T00:25:09.259 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-idna (3.3-1) ... 2024-05-06T00:25:09.450 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-06T00:25:09.538 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-06T00:25:09.551 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-06T00:25:09.576 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-06T00:25:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-06T00:25:09.797 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-06T00:25:09.987 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-06T00:25:10.065 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-06T00:25:10.238 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-06T00:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-06T00:25:10.514 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:10.559 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-06T00:25:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-06T00:25:10.665 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-06T00:25:10.794 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-06T00:25:10.892 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:10.976 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-06T00:25:11.002 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-06T00:25:11.020 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-argparse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:11.226 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-06T00:25:11.269 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-06T00:25:11.287 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-06T00:25:11.395 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-06T00:25:11.511 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-06T00:25:11.513 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-06T00:25:11.521 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-06T00:25:11.639 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-06T00:25:11.762 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-06T00:25:11.774 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-06T00:25:11.814 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-06T00:25:12.005 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-06T00:25:12.025 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-06T00:25:12.065 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-06T00:25:12.256 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-06T00:25:12.308 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:12.399 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-06T00:25:12.409 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-06T00:25:12.415 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-06T00:25:12.449 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-06T00:25:12.495 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-06T00:25:12.613 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-06T00:25:12.667 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-06T00:25:12.702 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-06T00:25:12.793 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-06T00:25:12.945 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-06T00:25:12.972 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-06T00:25:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-06T00:25:13.126 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-06T00:25:13.196 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-06T00:25:13.395 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-06T00:25:13.438 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-06T00:25:13.646 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-06T00:25:13.689 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-06T00:25:13.724 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-06T00:25:13.755 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-06T00:25:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-06T00:25:13.957 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-06T00:25:13.963 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-06T00:25:13.992 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-06T00:25:14.058 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-06T00:25:14.208 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-06T00:25:14.285 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up zip (3.0-12build2) ... 2024-05-06T00:25:14.309 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-06T00:25:14.326 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-06T00:25:14.411 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-06T00:25:14.452 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-06T00:25:14.703 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-06T00:25:14.728 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-06T00:25:14.962 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-06T00:25:15.004 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-06T00:25:15.046 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-06T00:25:15.088 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-06T00:25:15.255 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-06T00:25:15.297 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-06T00:25:15.348 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:15.381 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-06T00:25:15.423 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-06T00:25:15.474 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-06T00:25:15.507 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-06T00:25:15.558 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-06T00:25:15.604 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-06T00:25:15.800 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-06T00:25:15.850 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-06T00:25:15.851 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-06T00:25:15.991 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-06T00:25:16.085 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-06T00:25:16.210 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-06T00:25:16.352 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-06T00:25:16.361 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-06T00:25:16.436 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-06T00:25:16.512 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-06T00:25:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-06T00:25:16.722 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-06T00:25:16.779 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-06T00:25:16.780 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-06T00:25:16.981 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:17.039 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-06T00:25:17.063 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-06T00:25:17.107 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-06T00:25:17.385 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-06T00:25:17.408 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-06T00:25:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-06T00:25:17.674 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-06T00:25:17.735 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-06T00:25:17.786 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-06T00:25:17.800 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:18.003 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-06T00:25:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-06T00:25:18.076 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-idna (3.3-1) ... 2024-05-06T00:25:18.111 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-06T00:25:18.363 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-06T00:25:18.369 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-06T00:25:18.407 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-06T00:25:18.508 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librados2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:18.609 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:18.639 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-06T00:25:18.727 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-06T00:25:18.986 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-06T00:25:19.063 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:19.189 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libradosstriper1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:19.316 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-06T00:25:19.371 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-06T00:25:19.550 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-06T00:25:19.592 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librbd1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:19.639 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-06T00:25:19.718 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-modules-core (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:19.768 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:19.845 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:19.878 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-06T00:25:19.966 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-06T00:25:20.050 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-06T00:25:20.050 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-06T00:25:20.092 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librados2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:20.129 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-06T00:25:20.227 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:20.255 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-06T00:25:20.345 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-06T00:25:20.397 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-06T00:25:20.513 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs-dev (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:20.639 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rados (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:20.647 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:20.691 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-06T00:25:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librgw2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:20.773 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libradosstriper1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:20.890 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-06T00:25:20.916 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-06T00:25:20.942 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-06T00:25:21.168 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:21.192 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librbd1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:21.269 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-06T00:25:21.318 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-modules-core (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:21.444 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:21.587 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-06T00:25:21.640 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-06T00:25:21.641 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-06T00:25:21.846 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-06T00:25:22.003 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.104 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up rbd-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.105 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-06T00:25:22.129 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs-dev (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.221 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up rbd-nbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.255 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rados (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.323 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.356 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-06T00:25:22.381 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librgw2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.449 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cephfs (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.507 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-06T00:25:22.574 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:22.616 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-06T00:25:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-06T00:25:22.977 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-06T00:25:23.118 INFO:teuthology.orchestra.run.smithi078.stdout:Adding group ceph....done 2024-05-06T00:25:23.244 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-06T00:25:23.370 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-06T00:25:23.402 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user ceph....done 2024-05-06T00:25:23.580 INFO:teuthology.orchestra.run.smithi078.stdout:Setting system user ceph properties....done 2024-05-06T00:25:23.598 INFO:teuthology.orchestra.run.smithi078.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-06T00:25:23.663 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-06T00:25:23.702 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:23.734 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-06T00:25:23.812 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up rbd-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:23.921 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up rbd-nbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:23.964 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-06T00:25:24.030 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:24.090 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-06T00:25:24.190 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cephfs (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:24.232 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-06T00:25:24.315 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:24.350 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-06T00:25:24.467 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-06T00:25:24.539 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-06T00:25:24.710 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-test (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:24.761 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-06T00:25:24.836 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up radosgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:24.866 INFO:teuthology.orchestra.run.smithi186.stdout:Adding group ceph....done 2024-05-06T00:25:24.878 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-06T00:25:25.264 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-06T00:25:25.277 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user ceph....done 2024-05-06T00:25:25.297 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-06T00:25:25.298 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-06T00:25:25.411 INFO:teuthology.orchestra.run.smithi186.stdout:Setting system user ceph properties....done 2024-05-06T00:25:25.425 INFO:teuthology.orchestra.run.smithi186.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-06T00:25:25.522 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-06T00:25:25.559 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-06T00:25:25.649 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-06T00:25:25.771 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-base (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:25.869 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-06T00:25:25.908 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-06T00:25:26.009 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-06T00:25:26.089 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-06T00:25:26.302 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-06T00:25:26.316 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-test (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:26.442 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up radosgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:26.573 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mds (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:26.586 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-06T00:25:26.802 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-06T00:25:26.802 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-06T00:25:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-06T00:25:26.901 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-06T00:25:26.901 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-06T00:25:26.988 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-06T00:25:27.256 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-06T00:25:27.309 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:27.335 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-base (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:27.512 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-06T00:25:27.512 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-06T00:25:27.532 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-06T00:25:27.677 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-06T00:25:27.877 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-06T00:25:27.978 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-osd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:28.154 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mds (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:28.246 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-06T00:25:28.346 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-06T00:25:28.346 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-06T00:25:28.391 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-06T00:25:28.391 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-06T00:25:28.531 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-06T00:25:28.814 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:28.873 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:28.916 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-06T00:25:28.948 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:29.026 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librados2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:29.067 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-06T00:25:29.067 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-06T00:25:29.074 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mon (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:29.127 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:29.253 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-06T00:25:29.287 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-06T00:25:29.287 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-06T00:25:29.532 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-osd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:29.589 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:29.715 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libradosstriper1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:29.768 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:29.809 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-06T00:25:29.809 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-06T00:25:29.850 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-06T00:25:29.911 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.037 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-dashboard (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.101 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librbd1 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.172 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-volume (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-modules-core (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.250 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.303 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.385 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.518 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-06T00:25:30.518 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-06T00:25:30.525 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-06T00:25:30.536 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mon (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:30.634 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-06T00:25:30.740 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-06T00:25:30.741 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-06T00:25:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-06T00:25:31.079 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs-dev (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:31.204 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:31.205 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rados (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:31.331 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librgw2 (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:31.346 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:31.458 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-06T00:25:31.473 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-dashboard (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:31.480 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-06T00:25:31.615 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-volume (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:31.951 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-06T00:25:32.069 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-06T00:25:32.138 INFO:teuthology.orchestra.run.smithi078.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-06T00:25:32.143 DEBUG:teuthology.parallel:result is None 2024-05-06T00:25:32.221 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-06T00:25:32.619 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:32.745 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up rbd-fuse (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:32.871 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up rbd-nbd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:32.997 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:33.140 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cephfs (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:33.266 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-common (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:33.364 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-06T00:25:33.717 INFO:teuthology.orchestra.run.smithi146.stdout:Adding group ceph....done 2024-05-06T00:25:33.983 INFO:teuthology.orchestra.run.smithi186.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-06T00:25:33.988 DEBUG:teuthology.parallel:result is None 2024-05-06T00:25:34.127 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user ceph....done 2024-05-06T00:25:34.304 INFO:teuthology.orchestra.run.smithi146.stdout:Setting system user ceph properties....done 2024-05-06T00:25:34.319 INFO:teuthology.orchestra.run.smithi146.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-06T00:25:34.455 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-06T00:25:34.781 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-06T00:25:35.276 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-test (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:35.377 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up radosgw (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:35.823 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-06T00:25:35.824 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-06T00:25:36.304 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-base (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:36.646 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-06T00:25:37.180 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mds (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:37.406 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-06T00:25:37.406 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-06T00:25:37.915 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:38.144 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-06T00:25:38.144 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-06T00:25:38.650 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-osd (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:38.900 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-06T00:25:38.900 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-06T00:25:39.410 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:39.520 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:39.671 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mon (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:39.883 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-06T00:25:39.883 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-06T00:25:40.364 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-cephadm (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:40.517 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:40.618 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-dashboard (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:40.752 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-volume (19.0.0-3517-gb45ed440-1jammy) ... 2024-05-06T00:25:41.105 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-06T00:25:41.206 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-06T00:25:41.358 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-06T00:25:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-06T00:25:43.119 INFO:teuthology.orchestra.run.smithi146.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-06T00:25:43.124 DEBUG:teuthology.parallel:result is None 2024-05-06T00:25:43.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b45ed4400464709677363de54673bcce970bcb27 2024-05-06T00:25:43.275 DEBUG:teuthology.orchestra.run.smithi078:> dpkg-query -W -f '${Version}' ceph 2024-05-06T00:25:43.295 INFO:teuthology.orchestra.run.smithi078.stdout:19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:25:43.295 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:25:43.295 INFO:teuthology.task.install:The correct ceph version 19.0.0-3517-gb45ed440-1jammy is installed. 2024-05-06T00:25:43.297 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b45ed4400464709677363de54673bcce970bcb27 2024-05-06T00:25:43.508 DEBUG:teuthology.orchestra.run.smithi146:> dpkg-query -W -f '${Version}' ceph 2024-05-06T00:25:43.527 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:25:43.527 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:25:43.527 INFO:teuthology.task.install:The correct ceph version 19.0.0-3517-gb45ed440-1jammy is installed. 2024-05-06T00:25:43.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b45ed4400464709677363de54673bcce970bcb27 2024-05-06T00:25:43.666 DEBUG:teuthology.orchestra.run.smithi186:> dpkg-query -W -f '${Version}' ceph 2024-05-06T00:25:43.685 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:25:43.686 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3517-gb45ed440-1jammy 2024-05-06T00:25:43.686 INFO:teuthology.task.install:The correct ceph version 19.0.0-3517-gb45ed440-1jammy is installed. 2024-05-06T00:25:43.688 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-06T00:25:43.688 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:43.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-06T00:25:43.704 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:25:43.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-06T00:25:43.718 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:25:43.718 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-06T00:25:43.741 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-06T00:25:43.741 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:43.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-05-06T00:25:43.757 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-06T00:25:43.813 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:25:43.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-05-06T00:25:43.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-06T00:25:43.879 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:25:43.879 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-05-06T00:25:43.893 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-06T00:25:43.948 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-06T00:25:43.948 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:43.949 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-06T00:25:43.962 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-06T00:25:44.018 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:25:44.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-06T00:25:44.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-06T00:25:44.092 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:25:44.093 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-06T00:25:44.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-06T00:25:44.161 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-06T00:25:44.161 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:44.161 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-05-06T00:25:44.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-06T00:25:44.230 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:25:44.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-05-06T00:25:44.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-06T00:25:44.300 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:25:44.300 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-05-06T00:25:44.314 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-06T00:25:44.368 INFO:teuthology.run_tasks:Running task ceph... 2024-05-06T00:25:44.459 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-06T00:25:44.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2024-05-06T00:25:44.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-05-06T00:25:44.464 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-05-06T00:25:44.477 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-06T00:25:44.477 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-06T00:25:44.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-06T00:25:44.518 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-06T00:25:44.532 INFO:tasks.ceph:Creating extra log directories... 2024-05-06T00:25:44.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-06T00:25:44.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-06T00:25:44.574 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-06T00:25:44.596 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-06T00:25:44.596 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-06T00:25:44.596 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201', 'branch': 'wip-dis-testing', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nbd}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7692201', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'b45ed4400464709677363de54673bcce970bcb27'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'b45ed4400464709677363de54673bcce970bcb27'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': 'ed0ddec0117930fb8b53c34226cd9519ed394ff9'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 3217, 'sha1': 'b45ed4400464709677363de54673bcce970bcb27', 'sleep_before_teardown': 0, 'subset': '68/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-qa', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_ed0ddec0117930fb8b53c34226cd9519ed394ff9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': 'ed0ddec0117930fb8b53c34226cd9519ed394ff9', 'targets': {'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGgXUPNXOyznlxyLiumTfBns3jLkf8YE2UWAXwZi48NTiSQfPm/+LpLX7l8EX+Ypwyzcgs+sBU/B5RG+bdVINbA=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGeSaYtH1UYfLaevIGqmlmRzPXsS8Y2ayZ/xl5tqUhrAj9I0B3Oj06skZ8osS2ZvFR05O1WaNEqiuUcpJsK46QM=', 'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBV9KSFrlRHKzQoOzk+IqVYTf65hy4y5nncjEqhT7OIvDL2saCX7afDO2Eqj33fCjLHn2jedzbnz04DJNLR8xVI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'holebdy': 512, 'nbd': True, 'ops': 6000, 'punch_holes': True, 'readbdy': 512, 'truncbdy': 512, 'writebdy': 512}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-05_18:40:04', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-06T00:25:44.597 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-06T00:25:44.628 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-06T00:25:44.631 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-06T00:25:44.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2024-05-06T00:25:44.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-05-06T00:25:44.678 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-05-06T00:25:44.692 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:44.692 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-05-06T00:25:44.735 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-06T00:25:44.735 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-05-06T00:25:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-06T00:25:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T00:25:44.783 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-05-06T00:25:44.783 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T00:25:44.783 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-05-06 00:20:20.980241360 +0000 2024-05-06T00:25:44.783 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-05-06 00:20:20.748243434 +0000 2024-05-06T00:25:44.783 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-05-06 00:20:20.748243434 +0000 2024-05-06T00:25:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-05-06T00:25:44.784 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-06T00:25:44.841 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-05-06T00:25:44.841 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-05-06T00:25:44.841 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000385306 s, 1.3 MB/s 2024-05-06T00:25:44.842 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-06T00:25:44.892 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-05-06T00:25:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-06T00:25:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T00:25:44.939 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-05-06T00:25:44.939 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T00:25:44.939 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-05-06 00:20:21.504236680 +0000 2024-05-06T00:25:44.939 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-05-06 00:20:21.208239325 +0000 2024-05-06T00:25:44.939 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-05-06 00:20:21.208239325 +0000 2024-05-06T00:25:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-05-06T00:25:44.940 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-06T00:25:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-05-06T00:25:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-05-06T00:25:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000425601 s, 1.2 MB/s 2024-05-06T00:25:44.994 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-06T00:25:45.044 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-05-06T00:25:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-06T00:25:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T00:25:45.091 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-05-06T00:25:45.091 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T00:25:45.091 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-05-06 00:20:21.736234608 +0000 2024-05-06T00:25:45.091 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-05-06 00:20:21.732234644 +0000 2024-05-06T00:25:45.091 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-05-06 00:20:21.732234644 +0000 2024-05-06T00:25:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-05-06T00:25:45.091 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-06T00:25:45.145 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-05-06T00:25:45.145 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-05-06T00:25:45.145 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000440072 s, 1.2 MB/s 2024-05-06T00:25:45.146 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-06T00:25:45.196 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-05-06T00:25:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-06T00:25:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T00:25:45.243 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-05-06T00:25:45.243 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T00:25:45.243 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-05-06 00:20:22.464228106 +0000 2024-05-06T00:25:45.243 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-05-06 00:20:22.212230357 +0000 2024-05-06T00:25:45.243 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-05-06 00:20:22.212230357 +0000 2024-05-06T00:25:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-05-06T00:25:45.244 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-06T00:25:45.297 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-05-06T00:25:45.297 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-05-06T00:25:45.297 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000279842 s, 1.8 MB/s 2024-05-06T00:25:45.298 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-06T00:25:45.348 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-05-06T00:25:45.348 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:25:45.348 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-05-06T00:25:45.354 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-06T00:25:45.354 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-05-06T00:25:45.401 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-06T00:25:45.401 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T00:25:45.401 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-05-06T00:25:45.401 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T00:25:45.401 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-05-06 00:20:20.342383000 +0000 2024-05-06T00:25:45.401 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-06 00:20:20.094385215 +0000 2024-05-06T00:25:45.401 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-06 00:20:20.094385215 +0000 2024-05-06T00:25:45.401 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-05-06T00:25:45.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-06T00:25:45.455 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-05-06T00:25:45.455 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-05-06T00:25:45.455 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000454759 s, 1.1 MB/s 2024-05-06T00:25:45.457 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-06T00:25:45.506 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-05-06T00:25:45.552 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-06T00:25:45.552 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T00:25:45.552 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-05-06T00:25:45.553 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T00:25:45.553 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-05-06 00:20:20.866378320 +0000 2024-05-06T00:25:45.553 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-06 00:20:20.586380821 +0000 2024-05-06T00:25:45.553 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-06 00:20:20.586380821 +0000 2024-05-06T00:25:45.553 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-05-06T00:25:45.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-06T00:25:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-05-06T00:25:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-05-06T00:25:45.607 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000396692 s, 1.3 MB/s 2024-05-06T00:25:45.608 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-06T00:25:45.657 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-05-06T00:25:45.706 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-06T00:25:45.706 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T00:25:45.706 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-05-06T00:25:45.706 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T00:25:45.706 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-05-06 00:20:21.362373890 +0000 2024-05-06T00:25:45.707 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-06 00:20:21.110376141 +0000 2024-05-06T00:25:45.707 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-06 00:20:21.110376141 +0000 2024-05-06T00:25:45.707 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-05-06T00:25:45.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-06T00:25:45.759 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-05-06T00:25:45.759 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-05-06T00:25:45.759 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000434974 s, 1.2 MB/s 2024-05-06T00:25:45.760 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-06T00:25:45.810 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-05-06T00:25:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-06T00:25:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T00:25:45.856 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-05-06T00:25:45.857 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T00:25:45.857 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-05-06 00:20:21.862369423 +0000 2024-05-06T00:25:45.857 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-06 00:20:21.594371817 +0000 2024-05-06T00:25:45.857 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-06 00:20:21.594371817 +0000 2024-05-06T00:25:45.857 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-05-06T00:25:45.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-06T00:25:45.911 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-05-06T00:25:45.911 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-05-06T00:25:45.911 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000382727 s, 1.3 MB/s 2024-05-06T00:25:45.912 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-06T00:25:45.962 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-05-06T00:25:45.962 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-05-06T00:25:45.962 INFO:tasks.ceph:Generating config... 2024-05-06T00:25:45.963 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-06T00:25:45.963 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-05-06T00:25:45.963 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-06T00:25:45.963 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-06T00:25:45.964 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-06T00:25:45.964 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-06T00:25:45.964 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-06T00:25:45.964 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-06T00:25:45.964 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-06T00:25:45.965 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-06T00:25:45.965 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-06T00:25:45.965 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-05-06T00:25:45.965 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-06T00:25:45.965 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-05-06T00:25:45.965 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-06T00:25:45.965 INFO:tasks.ceph:Setting up mon.a... 2024-05-06T00:25:45.965 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-06T00:25:45.998 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.keyring 2024-05-06T00:25:46.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-06T00:25:46.098 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-06T00:25:46.157 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.78'), ('mon.c', '[v2:172.21.15.78:3301,v1:172.21.15.78:6790]'), ('mon.b', '172.21.15.146')] 2024-05-06T00:25:46.158 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.146', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-05-06T00:25:46.159 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:46.159 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-06T00:25:46.206 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.78 --addv c '[v2:172.21.15.78:3301,v1:172.21.15.78:6790]' --add b 172.21.15.146 --print /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:25:46.294 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:25:46.294 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: generated fsid b583c614-b2ca-4769-90c0-44726bf69b34 2024-05-06T00:25:46.294 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = quincy 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 0 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:fsid b583c614-b2ca-4769-90c0-44726bf69b34 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:last_changed 2024-05-06T00:25:46.294954+0000 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-05-06T00:25:46.294954+0000 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:min_mon_release 17 (quincy) 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:election_strategy: 1 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-05-06T00:25:46.295 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-06T00:25:46.296 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-06T00:25:46.343 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b583c614-b2ca-4769-90c0-44726bf69b34... 2024-05-06T00:25:46.344 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-06T00:25:46.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-06T00:25:46.390 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-06T00:25:46.415 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-05-06T00:25:46.415 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2024-05-06T00:25:46.415 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-06T00:25:46.415 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2024-05-06T00:25:46.415 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-06T00:25:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = "172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.146" 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: mon client directed command retry = 5 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on pool no app = False 2024-05-06T00:25:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject socket failures = 5000 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = b583c614-b2ca-4769-90c0-44726bf69b34 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug trim objects = true 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2024-05-06T00:25:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: bluefs allocator = hybrid 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore allocator = hybrid 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 1/20 2024-05-06T00:25:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 1/20 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 4/10 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: osd blocked scrub grace period = 3600 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: osd max markdown count = 1000 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-05-06T00:25:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-05-06T00:25:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.c] 2024-05-06T00:25:46.425 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.b] 2024-05-06T00:25:46.426 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-05-06T00:25:46.426 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-05-06T00:25:46.426 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-06T00:25:46.426 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-05-06T00:25:46.426 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.426 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-05-06T00:25:46.426 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-05-06T00:25:46.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.146" 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon client directed command retry = 5 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on pool no app = False 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject socket failures = 5000 2024-05-06T00:25:46.429 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = b583c614-b2ca-4769-90c0-44726bf69b34 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-05-06T00:25:46.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug trim objects = true 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-05-06T00:25:46.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: bluefs allocator = hybrid 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = hybrid 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1/20 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1/20 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 4/10 2024-05-06T00:25:46.432 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd blocked scrub grace period = 3600 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd max markdown count = 1000 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-06T00:25:46.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-05-06T00:25:46.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-05-06T00:25:46.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-05-06T00:25:46.436 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-05-06T00:25:46.437 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-06T00:25:46.438 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.146" 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: mon client directed command retry = 5 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on pool no app = False 2024-05-06T00:25:46.439 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject socket failures = 5000 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = b583c614-b2ca-4769-90c0-44726bf69b34 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-05-06T00:25:46.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.441 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: bluefs allocator = hybrid 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = hybrid 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-05-06T00:25:46.442 INFO:teuthology.orchestra.run.smithi146.stdout: osd blocked scrub grace period = 3600 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd max markdown count = 1000 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.443 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-05-06T00:25:46.444 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-05-06T00:25:46.445 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-06T00:25:46.446 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-06T00:25:46.446 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-05-06T00:25:46.446 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-05-06T00:25:46.446 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-05-06T00:25:46.448 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-06T00:25:46.448 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-06T00:25:46.538 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-06T00:25:46.538 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:46.538 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-06T00:25:46.543 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:46.543 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-06T00:25:46.594 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:25:46.595 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:46.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-06T00:25:46.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-06T00:25:46.656 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:25:46.657 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:25:46.706 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:25:46.706 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:25:46.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-06T00:25:46.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-06T00:25:46.728 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:25:46.728 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:25:46.777 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-05-06T00:25:46.777 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:25:46.777 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-06T00:25:46.777 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-06T00:25:46.800 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:25:46.800 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:25:46.846 INFO:tasks.ceph:Setting up mon nodes... 2024-05-06T00:25:46.846 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-06T00:25:46.846 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-06T00:25:46.889 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-06T00:25:46.902 INFO:tasks.ceph:Setting up mds nodes... 2024-05-06T00:25:46.902 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-06T00:25:46.902 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-06T00:25:46.938 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-06T00:25:46.960 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-06T00:25:46.960 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-05-06T00:25:46.960 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-06T00:25:46.975 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-05-06T00:25:46.975 INFO:tasks.ceph:role: osd.0 2024-05-06T00:25:46.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:25:46.975 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-06T00:25:47.038 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-06T00:25:47.038 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-06T00:25:47.038 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-06T00:25:47.039 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-06T00:25:47.039 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-06T00:25:47.039 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-05-06T00:25:47.039 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-06T00:25:47.039 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-06T00:25:47.039 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-06T00:25:47.039 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-06T00:25:47.532 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-05-06T00:25:47.534 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-05-06T00:25:47.534 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-06T00:25:47.699 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-06T00:25:47.710 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-05-06T00:25:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:25:47.711 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-06T00:25:47.766 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-05-06T00:25:47.766 INFO:tasks.ceph:role: osd.1 2024-05-06T00:25:47.766 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:25:47.766 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-06T00:25:47.830 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-06T00:25:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-06T00:25:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-06T00:25:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-06T00:25:47.830 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-06T00:25:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-05-06T00:25:47.831 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-06T00:25:47.831 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-06T00:25:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-06T00:25:47.831 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-06T00:25:48.278 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-05-06T00:25:48.281 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-05-06T00:25:48.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-06T00:25:48.366 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-06T00:25:48.423 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-05-06T00:25:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:25:48.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-06T00:25:48.477 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-05-06T00:25:48.477 INFO:tasks.ceph:role: osd.2 2024-05-06T00:25:48.478 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:25:48.478 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-06T00:25:48.542 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-06T00:25:48.542 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-06T00:25:48.542 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-06T00:25:48.542 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-06T00:25:48.543 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-06T00:25:48.543 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-05-06T00:25:48.543 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-06T00:25:48.543 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-06T00:25:48.543 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-06T00:25:48.543 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-06T00:25:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-05-06T00:25:49.014 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-05-06T00:25:49.014 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-06T00:25:49.148 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-06T00:25:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:25:49.160 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-05-06T00:25:49.161 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:25:49.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:49.274+0000 7f0ded8c77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-06T00:25:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:49.278+0000 7f0ded8c77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-06T00:25:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:49.278+0000 7f0ded8c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-06T00:25:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:49.278+0000 7f0ded8c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-06T00:25:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:49.278+0000 7f0ded8c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-06T00:25:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:49.286+0000 7f0ded8c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-06T00:25:52.269 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T00:25:52.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:25:52.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:52.362+0000 7f0197e657c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-06T00:25:52.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:52.362+0000 7f0197e657c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-06T00:25:52.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:52.362+0000 7f0197e657c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-06T00:25:52.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:52.362+0000 7f0197e657c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-06T00:25:52.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:52.362+0000 7f0197e657c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-06T00:25:52.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:52.362+0000 7f0197e657c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-06T00:25:55.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T00:25:55.407 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:25:55.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:55.486+0000 7f33b5e257c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-06T00:25:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:55.486+0000 7f33b5e257c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-06T00:25:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:55.486+0000 7f33b5e257c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-06T00:25:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:55.486+0000 7f33b5e257c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-06T00:25:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:55.486+0000 7f33b5e257c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-06T00:25:55.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:25:55.490+0000 7f33b5e257c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-06T00:25:58.478 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T00:25:58.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-06T00:25:58.550 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-05-06T00:25:58.550 INFO:tasks.ceph:role: osd.3 2024-05-06T00:25:58.550 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:25:58.551 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-06T00:25:58.614 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-06T00:25:58.614 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-06T00:25:58.614 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-06T00:25:58.614 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-06T00:25:58.614 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-06T00:25:58.614 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-05-06T00:25:58.614 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-06T00:25:58.614 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-06T00:25:58.615 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-06T00:25:58.615 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-06T00:25:59.069 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-05-06T00:25:59.071 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-05-06T00:25:59.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-05-06T00:25:59.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-06T00:25:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:25:59.210 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2024-05-06T00:25:59.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-06T00:25:59.263 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-05-06T00:25:59.263 INFO:tasks.ceph:role: osd.4 2024-05-06T00:25:59.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:25:59.263 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-06T00:25:59.324 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-06T00:25:59.324 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-06T00:25:59.324 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-06T00:25:59.324 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-06T00:25:59.324 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-06T00:25:59.324 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-05-06T00:25:59.324 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-06T00:25:59.324 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-06T00:25:59.325 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-06T00:25:59.325 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-06T00:25:59.791 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-05-06T00:25:59.793 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-05-06T00:25:59.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-05-06T00:25:59.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-06T00:25:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:25:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2024-05-06T00:25:59.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-06T00:26:00.011 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-05-06T00:26:00.011 INFO:tasks.ceph:role: osd.5 2024-05-06T00:26:00.012 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:26:00.012 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-06T00:26:00.070 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-06T00:26:00.071 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-06T00:26:00.561 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-05-06T00:26:00.563 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-05-06T00:26:00.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-05-06T00:26:00.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-06T00:26:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:26:00.715 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2024-05-06T00:26:00.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:26:00.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:00.825+0000 7fad7f17d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-06T00:26:00.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:00.825+0000 7fad7f17d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-06T00:26:00.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:00.825+0000 7fad7f17d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-06T00:26:00.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:00.825+0000 7fad7f17d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-06T00:26:00.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:00.825+0000 7fad7f17d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-06T00:26:00.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:00.825+0000 7fad7f17d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-06T00:26:03.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T00:26:03.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:26:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:03.929+0000 7f1c30ec47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-06T00:26:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:03.929+0000 7f1c30ec47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-06T00:26:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:03.929+0000 7f1c30ec47c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-06T00:26:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:03.929+0000 7f1c30ec47c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-06T00:26:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:03.929+0000 7f1c30ec47c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-06T00:26:03.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:03.929+0000 7f1c30ec47c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-06T00:26:06.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T00:26:06.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:26:07.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:07.045+0000 7fdc6cdbb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-06T00:26:07.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:07.045+0000 7fdc6cdbb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-06T00:26:07.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:07.045+0000 7fdc6cdbb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-06T00:26:07.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:07.045+0000 7fdc6cdbb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-06T00:26:07.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:07.045+0000 7fdc6cdbb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-06T00:26:07.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-06T00:26:07.045+0000 7fdc6cdbb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-06T00:26:10.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T00:26:10.073 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-06T00:26:10.073 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:26:10.073 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-06T00:26:10.088 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:26:10.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-06T00:26:10.141 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:26:10.141 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-06T00:26:10.195 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:26:10.195 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-06T00:26:10.209 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:26:10.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-06T00:26:10.263 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:26:10.263 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-06T00:26:10.315 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:26:10.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-06T00:26:10.367 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-06T00:26:10.368 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-06T00:26:10.374 INFO:tasks.ceph:Adding keys to all mons... 2024-05-06T00:26:10.374 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-06T00:26:10.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-06T00:26:10.415 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2024-05-06T00:26:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCNIzhmJednEBAAyFSYe/TIQTRpsB2emgmDbw== 2024-05-06T00:26:10.415 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2024-05-06T00:26:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCQIzhmDqG5FRAAJGnp9r0/KbfChOkl4LdD/g== 2024-05-06T00:26:10.415 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2024-05-06T00:26:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCTIzhmh1wuHRAAGSkWf0Mi6a33J87VM0CV9g== 2024-05-06T00:26:10.415 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCKIzhmh6koNRAA3aeW3Iy7/c7G1khhUwjbdA== 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.3] 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCYIzhmxS1LMRAAysJGVFLuqltobeaygP2tYA== 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.4] 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCbIzhmOyaWNxAAbWqQD87dj0Y6cCbsOS0fZw== 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.5] 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCfIzhmVEXjAhAAIbYQYeaL5Eclc1IkQK5HrQ== 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-05-06T00:26:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCKIzhm8iIXOBAALX+NNtPYjr01dh+CR2wSDw== 2024-05-06T00:26:10.422 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-05-06T00:26:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCNIzhmJednEBAAyFSYe/TIQTRpsB2emgmDbw== 2024-05-06T00:26:10.423 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-05-06T00:26:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCQIzhmDqG5FRAAJGnp9r0/KbfChOkl4LdD/g== 2024-05-06T00:26:10.423 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-05-06T00:26:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCTIzhmh1wuHRAAGSkWf0Mi6a33J87VM0CV9g== 2024-05-06T00:26:10.425 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-05-06T00:26:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCKIzhmh6koNRAA3aeW3Iy7/c7G1khhUwjbdA== 2024-05-06T00:26:10.425 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-05-06T00:26:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCYIzhmxS1LMRAAysJGVFLuqltobeaygP2tYA== 2024-05-06T00:26:10.425 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-05-06T00:26:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCbIzhmOyaWNxAAbWqQD87dj0Y6cCbsOS0fZw== 2024-05-06T00:26:10.426 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-05-06T00:26:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCfIzhmVEXjAhAAIbYQYeaL5Eclc1IkQK5HrQ== 2024-05-06T00:26:10.426 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-05-06T00:26:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCKIzhm8iIXOBAALX+NNtPYjr01dh+CR2wSDw== 2024-05-06T00:26:10.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.644 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-06T00:26:10.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-06T00:26:10.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.737 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.899 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-06T00:26:10.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-06T00:26:10.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-06T00:26:11.076 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-06T00:26:11.077 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-06T00:26:11.089 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-06T00:26:11.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-06T00:26:11.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-06T00:26:11.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-06T00:26:11.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-06T00:26:11.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-06T00:26:11.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-06T00:26:11.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-06T00:26:11.462 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:26:11.464 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-06T00:26:11.508 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-06T00:26:11.508 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-06T00:26:11.508 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-06T00:26:11.511 INFO:tasks.ceph.mon.a:Started 2024-05-06T00:26:11.511 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-06T00:26:11.511 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-06T00:26:11.514 INFO:tasks.ceph.mon.c:Started 2024-05-06T00:26:11.514 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-06T00:26:11.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-06T00:26:11.553 INFO:tasks.ceph.mon.b:Started 2024-05-06T00:26:11.553 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-06T00:26:11.553 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-06T00:26:11.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-06T00:26:11.556 INFO:tasks.ceph.mgr.x:Started 2024-05-06T00:26:11.556 INFO:tasks.ceph:Setting crush tunables to default 2024-05-06T00:26:11.556 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd crush tunables default 2024-05-06T00:26:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-05-06T00:26:17.063 INFO:tasks.ceph:check_enable_crimson: False 2024-05-06T00:26:17.063 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-06T00:26:17.063 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:26:17.063 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-06T00:26:17.075 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:26:17.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-06T00:26:17.131 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-06T00:26:17.131 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-06T00:26:17.183 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:26:17.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-06T00:26:17.198 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:26:17.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-06T00:26:17.252 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-06T00:26:17.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-06T00:26:17.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new d31b9a6a-864b-4c24-99db-8cc6cf3a5c24 0 2024-05-06T00:26:17.569 INFO:teuthology.orchestra.run.smithi146.stdout:0 2024-05-06T00:26:17.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 86509826-0e53-4678-8260-17c97c05af54 1 2024-05-06T00:26:17.801 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-05-06T00:26:17.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 8973b524-521e-41c0-87f2-85cb390fbfeb 2 2024-05-06T00:26:17.855 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:17.853+0000 7f8a6c564180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T00:26:18.032 INFO:teuthology.orchestra.run.smithi146.stdout:2 2024-05-06T00:26:18.050 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 3b0d07bd-132e-48ff-83ac-7182eee05df9 3 2024-05-06T00:26:18.316 INFO:teuthology.orchestra.run.smithi146.stdout:3 2024-05-06T00:26:18.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2 4 2024-05-06T00:26:18.394 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:18.393+0000 7f8a6c564180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T00:26:18.538 INFO:teuthology.orchestra.run.smithi146.stdout:4 2024-05-06T00:26:18.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 34b61f5f-e407-4414-8bc5-11263bbe261a 5 2024-05-06T00:26:18.609 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:18.609+0000 7f8a6c564180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T00:26:18.758 INFO:teuthology.orchestra.run.smithi146.stdout:5 2024-05-06T00:26:18.775 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-06T00:26:18.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-06T00:26:18.778 INFO:tasks.ceph.osd.0:Started 2024-05-06T00:26:18.778 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-06T00:26:18.778 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-06T00:26:18.780 INFO:tasks.ceph.osd.1:Started 2024-05-06T00:26:18.781 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-06T00:26:18.781 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-06T00:26:18.784 INFO:tasks.ceph.osd.2:Started 2024-05-06T00:26:18.784 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-06T00:26:18.784 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-06T00:26:18.787 INFO:tasks.ceph.osd.3:Started 2024-05-06T00:26:18.787 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-06T00:26:18.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-06T00:26:18.789 INFO:tasks.ceph.osd.4:Started 2024-05-06T00:26:18.789 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-06T00:26:18.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-06T00:26:18.791 INFO:tasks.ceph.osd.5:Started 2024-05-06T00:26:18.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T00:26:18.981 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:18.981 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:26:18.750504+0000","last_up_change":"0.000000","last_in_change":"2024-05-06T00:26:18.750504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:26:18.998 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-06T00:26:18.998 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-06T00:26:19.168 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.169+0000 7f8a6c564180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T00:26:19.222 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.221+0000 7f8a6c564180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T00:26:19.274 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.273+0000 7f8a6c564180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T00:26:19.350 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.349+0000 7f8a6c564180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T00:26:19.389 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:19.389+0000 7f64dba8d7c0 -1 Falling back to public interface 2024-05-06T00:26:19.395 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:19.394+0000 7f8d4b1ee7c0 -1 Falling back to public interface 2024-05-06T00:26:19.396 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:19.394+0000 7faa921007c0 -1 Falling back to public interface 2024-05-06T00:26:19.396 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:19.394+0000 7f8d4a1ae640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-05-06T00:26:19.402 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.401+0000 7f8a6c564180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T00:26:19.408 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:19.409+0000 7f94cd9e57c0 -1 Falling back to public interface 2024-05-06T00:26:19.409 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:19.409+0000 7fe807a837c0 -1 Falling back to public interface 2024-05-06T00:26:19.422 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:19.422+0000 7f1eb33207c0 -1 Falling back to public interface 2024-05-06T00:26:19.482 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.481+0000 7f8a6c564180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T00:26:19.535 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.533+0000 7f8a6c564180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T00:26:19.601 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-06T00:26:19.682 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.681+0000 7f8a6c564180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T00:26:19.744 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:19.745+0000 7f8a6c564180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T00:26:19.791 INFO:teuthology.misc.health.smithi078.stdout: 2024-05-06T00:26:19.791 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":8,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:26:18.750504+0000","last_up_change":"0.000000","last_in_change":"2024-05-06T00:26:18.750504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:26:19.807 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-05-06T00:26:20.407 INFO:tasks.ceph.mgr.x.smithi146.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-06T00:26:20.408 INFO:tasks.ceph.mgr.x.smithi146.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-06T00:26:20.408 INFO:tasks.ceph.mgr.x.smithi146.stderr: from numpy import show_config as show_numpy_config 2024-05-06T00:26:20.476 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:20.473+0000 7f8a6c564180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T00:26:20.657 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:20.657+0000 7f8a6c564180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T00:26:20.832 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:20.833+0000 7f8a6c564180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T00:26:20.902 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:20.901+0000 7f8a6c564180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T00:26:21.049 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:21.049+0000 7f8a6c564180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T00:26:21.667 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:21.665+0000 7f8a6c564180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T00:26:21.742 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:21.741+0000 7f8a6c564180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T00:26:21.923 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:21.921+0000 7f8a6c564180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T00:26:21.996 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:21.993+0000 7f8a6c564180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T00:26:22.294 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:22.293+0000 7f8a6c564180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T00:26:22.356 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:22.353+0000 7f8a6c564180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T00:26:23.073 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:23.070+0000 7f8d4b1ee7c0 -1 osd.1 0 log_to_monitors true 2024-05-06T00:26:23.184 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:23.185+0000 7f64dba8d7c0 -1 osd.5 0 log_to_monitors true 2024-05-06T00:26:23.189 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:23.186+0000 7faa921007c0 -1 osd.2 0 log_to_monitors true 2024-05-06T00:26:23.234 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:23.233+0000 7fe807a837c0 -1 osd.3 0 log_to_monitors true 2024-05-06T00:26:23.269 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:23.266+0000 7f1eb33207c0 -1 osd.0 0 log_to_monitors true 2024-05-06T00:26:23.296 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:23.297+0000 7f94cd9e57c0 -1 osd.4 0 log_to_monitors true 2024-05-06T00:26:23.401 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:23.401+0000 7f8a130a8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-06T00:26:23.403 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-05-06T00:26:23.401+0000 7f8a130a8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-06T00:26:26.373 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:26.370+0000 7f8d47197640 -1 osd.1 0 waiting for initial osdmap 2024-05-06T00:26:26.384 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:26.385+0000 7fe803a2c640 -1 osd.3 0 waiting for initial osdmap 2024-05-06T00:26:26.392 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:26.390+0000 7f1eaf2c9640 -1 osd.0 0 waiting for initial osdmap 2024-05-06T00:26:26.395 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:26.394+0000 7f8d3efcb640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T00:26:26.413 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-06T00:26:26.414 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:26.413+0000 7fe7fb860640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T00:26:26.414 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:26.414+0000 7f1ea70fd640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T00:26:26.420 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:26.418+0000 7faa8e0a9640 -1 osd.2 0 waiting for initial osdmap 2024-05-06T00:26:26.429 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:26.429+0000 7f64d7a36640 -1 osd.5 0 waiting for initial osdmap 2024-05-06T00:26:26.435 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:26.433+0000 7f94c998e640 -1 osd.4 0 waiting for initial osdmap 2024-05-06T00:26:26.438 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:26.438+0000 7faa85edd640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T00:26:26.456 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:26.457+0000 7f64cf86a640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T00:26:26.473 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:26.473+0000 7f94c17c2640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T00:26:26.697 INFO:teuthology.misc.health.smithi078.stdout: 2024-05-06T00:26:26.697 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":10,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:26:24.474691+0000","last_up_change":"0.000000","last_in_change":"2024-05-06T00:26:18.750504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:26:26.715 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-05-06T00:26:33.322 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-06T00:26:33.619 INFO:teuthology.misc.health.smithi078.stdout: 2024-05-06T00:26:33.619 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":14,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:26:30.594530+0000","last_up_change":"2024-05-06T00:26:27.394947+0000","last_in_change":"2024-05-06T00:26:18.750504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T00:26:28.436494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6817","nonce":1427564096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6819","nonce":1427564096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6823","nonce":1427564096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6821","nonce":1427564096}]},"public_addr":"172.21.15.78:6817/1427564096","cluster_addr":"172.21.15.78:6819/1427564096","heartbeat_back_addr":"172.21.15.78:6823/1427564096","heartbeat_front_addr":"172.21.15.78:6821/1427564096","state":["exists","up"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6801","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6803","nonce":2663808859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6805","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6807","nonce":2663808859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6815","nonce":2663808859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6811","nonce":2663808859}]},"public_addr":"172.21.15.78:6803/2663808859","cluster_addr":"172.21.15.78:6807/2663808859","heartbeat_back_addr":"172.21.15.78:6815/2663808859","heartbeat_front_addr":"172.21.15.78:6811/2663808859","state":["exists","up"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6802","nonce":385938386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6806","nonce":385938386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6814","nonce":385938386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6810","nonce":385938386}]},"public_addr":"172.21.15.78:6802/385938386","cluster_addr":"172.21.15.78:6806/385938386","heartbeat_back_addr":"172.21.15.78:6814/385938386","heartbeat_front_addr":"172.21.15.78:6810/385938386","state":["exists","up"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6812","nonce":4178038317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6816","nonce":4178038317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6822","nonce":4178038317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6817","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6819","nonce":4178038317}]},"public_addr":"172.21.15.146:6812/4178038317","cluster_addr":"172.21.15.146:6816/4178038317","heartbeat_back_addr":"172.21.15.146:6822/4178038317","heartbeat_front_addr":"172.21.15.146:6819/4178038317","state":["exists","up"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6809","nonce":1052555350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6813","nonce":1052555350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6823","nonce":1052555350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6818","nonce":1052555350}]},"public_addr":"172.21.15.146:6809/1052555350","cluster_addr":"172.21.15.146:6813/1052555350","heartbeat_back_addr":"172.21.15.146:6823/1052555350","heartbeat_front_addr":"172.21.15.146:6818/1052555350","state":["exists","up"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6801","nonce":2019146845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6803","nonce":2019146845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6807","nonce":2019146845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6805","nonce":2019146845}]},"public_addr":"172.21.15.146:6801/2019146845","cluster_addr":"172.21.15.146:6803/2019146845","heartbeat_back_addr":"172.21.15.146:6807/2019146845","heartbeat_front_addr":"172.21.15.146:6805/2019146845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.276988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.054473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.154558+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.284746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.259847+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.180375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:26:33.636 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-05-06T00:26:33.636 INFO:tasks.ceph:Creating RBD pool 2024-05-06T00:26:33.637 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-06T00:26:34.626 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'rbd' created 2024-05-06T00:26:34.651 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-06T00:26:35.636 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-06T00:26:35.662 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-06T00:26:35.662 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-06T00:26:35.662 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-06T00:26:35.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-06T00:26:35.663 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-06T00:26:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:36.020 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"flags":0,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":350800091},{"type":"v1","addr":"172.21.15.146:6825","nonce":350800091}]},"active_addr":"172.21.15.146:6825/350800091","active_change":"2024-05-06T00:26:22.379656+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3484265739}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2257567263}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3113746738}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":4016804612}]}]} 2024-05-06T00:26:36.025 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-06T00:26:36.025 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-06T00:26:36.025 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T00:26:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:36.308 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":16,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:26:35.622405+0000","last_up_change":"2024-05-06T00:26:27.394947+0000","last_in_change":"2024-05-06T00:26:18.750504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T00:26:28.436494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-06T00:26:33.941780+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6817","nonce":1427564096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6819","nonce":1427564096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6823","nonce":1427564096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6821","nonce":1427564096}]},"public_addr":"172.21.15.78:6817/1427564096","cluster_addr":"172.21.15.78:6819/1427564096","heartbeat_back_addr":"172.21.15.78:6823/1427564096","heartbeat_front_addr":"172.21.15.78:6821/1427564096","state":["exists","up"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6801","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6803","nonce":2663808859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6805","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6807","nonce":2663808859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6815","nonce":2663808859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6811","nonce":2663808859}]},"public_addr":"172.21.15.78:6803/2663808859","cluster_addr":"172.21.15.78:6807/2663808859","heartbeat_back_addr":"172.21.15.78:6815/2663808859","heartbeat_front_addr":"172.21.15.78:6811/2663808859","state":["exists","up"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6802","nonce":385938386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6806","nonce":385938386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6814","nonce":385938386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6810","nonce":385938386}]},"public_addr":"172.21.15.78:6802/385938386","cluster_addr":"172.21.15.78:6806/385938386","heartbeat_back_addr":"172.21.15.78:6814/385938386","heartbeat_front_addr":"172.21.15.78:6810/385938386","state":["exists","up"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6812","nonce":4178038317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6816","nonce":4178038317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6822","nonce":4178038317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6817","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6819","nonce":4178038317}]},"public_addr":"172.21.15.146:6812/4178038317","cluster_addr":"172.21.15.146:6816/4178038317","heartbeat_back_addr":"172.21.15.146:6822/4178038317","heartbeat_front_addr":"172.21.15.146:6819/4178038317","state":["exists","up"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6809","nonce":1052555350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6813","nonce":1052555350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6823","nonce":1052555350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6818","nonce":1052555350}]},"public_addr":"172.21.15.146:6809/1052555350","cluster_addr":"172.21.15.146:6813/1052555350","heartbeat_back_addr":"172.21.15.146:6823/1052555350","heartbeat_front_addr":"172.21.15.146:6818/1052555350","state":["exists","up"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6801","nonce":2019146845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6803","nonce":2019146845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6807","nonce":2019146845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6805","nonce":2019146845}]},"public_addr":"172.21.15.146:6801/2019146845","cluster_addr":"172.21.15.146:6803/2019146845","heartbeat_back_addr":"172.21.15.146:6807/2019146845","heartbeat_front_addr":"172.21.15.146:6805/2019146845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.276988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.054473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.154558+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.284746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.259847+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.180375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:26:36.325 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-06T00:26:36.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T00:26:36.620 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:36.620 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":16,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:26:35.622405+0000","last_up_change":"2024-05-06T00:26:27.394947+0000","last_in_change":"2024-05-06T00:26:18.750504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T00:26:28.436494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-06T00:26:33.941780+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6817","nonce":1427564096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6819","nonce":1427564096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6823","nonce":1427564096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6821","nonce":1427564096}]},"public_addr":"172.21.15.78:6817/1427564096","cluster_addr":"172.21.15.78:6819/1427564096","heartbeat_back_addr":"172.21.15.78:6823/1427564096","heartbeat_front_addr":"172.21.15.78:6821/1427564096","state":["exists","up"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6801","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6803","nonce":2663808859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6805","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6807","nonce":2663808859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6815","nonce":2663808859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6811","nonce":2663808859}]},"public_addr":"172.21.15.78:6803/2663808859","cluster_addr":"172.21.15.78:6807/2663808859","heartbeat_back_addr":"172.21.15.78:6815/2663808859","heartbeat_front_addr":"172.21.15.78:6811/2663808859","state":["exists","up"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6802","nonce":385938386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6806","nonce":385938386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6814","nonce":385938386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6810","nonce":385938386}]},"public_addr":"172.21.15.78:6802/385938386","cluster_addr":"172.21.15.78:6806/385938386","heartbeat_back_addr":"172.21.15.78:6814/385938386","heartbeat_front_addr":"172.21.15.78:6810/385938386","state":["exists","up"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6812","nonce":4178038317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6816","nonce":4178038317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6822","nonce":4178038317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6817","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6819","nonce":4178038317}]},"public_addr":"172.21.15.146:6812/4178038317","cluster_addr":"172.21.15.146:6816/4178038317","heartbeat_back_addr":"172.21.15.146:6822/4178038317","heartbeat_front_addr":"172.21.15.146:6819/4178038317","state":["exists","up"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6809","nonce":1052555350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6813","nonce":1052555350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6823","nonce":1052555350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":1052555350},{"type":"v1","addr":"172.21.15.146:6818","nonce":1052555350}]},"public_addr":"172.21.15.146:6809/1052555350","cluster_addr":"172.21.15.146:6813/1052555350","heartbeat_back_addr":"172.21.15.146:6823/1052555350","heartbeat_front_addr":"172.21.15.146:6818/1052555350","state":["exists","up"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6801","nonce":2019146845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6803","nonce":2019146845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6807","nonce":2019146845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6805","nonce":2019146845}]},"public_addr":"172.21.15.146:6801/2019146845","cluster_addr":"172.21.15.146:6803/2019146845","heartbeat_back_addr":"172.21.15.146:6807/2019146845","heartbeat_front_addr":"172.21.15.146:6805/2019146845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.276988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.054473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.154558+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.284746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.259847+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.180375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:26:36.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-06T00:26:36.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-06T00:26:36.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-06T00:26:36.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-06T00:26:36.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-06T00:26:36.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-06T00:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:47244640262 2024-05-06T00:26:36.803 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-06T00:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-05-06T00:26:36.814 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-06T00:26:36.854 INFO:teuthology.orchestra.run.smithi078.stdout:47244640260 2024-05-06T00:26:36.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-06T00:26:36.875 INFO:teuthology.orchestra.run.smithi078.stdout:47244640260 2024-05-06T00:26:36.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-06T00:26:36.882 INFO:teuthology.orchestra.run.smithi078.stdout:47244640260 2024-05-06T00:26:36.882 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-06T00:26:36.883 INFO:teuthology.orchestra.run.smithi078.stdout:47244640262 2024-05-06T00:26:36.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-06T00:26:37.225 INFO:teuthology.orchestra.run.smithi078.stdout:47244640259 2024-05-06T00:26:37.238 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-05-06T00:26:37.241 INFO:teuthology.orchestra.run.smithi078.stdout:47244640259 2024-05-06T00:26:37.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-05-06T00:26:37.261 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.3 2024-05-06T00:26:37.268 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.1 2024-05-06T00:26:37.271 INFO:teuthology.orchestra.run.smithi078.stdout:47244640260 2024-05-06T00:26:37.285 INFO:teuthology.orchestra.run.smithi078.stdout:47244640259 2024-05-06T00:26:37.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.2 2024-05-06T00:26:37.302 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.4 2024-05-06T00:26:37.344 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-05-06T00:26:37.360 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.5 2024-05-06T00:26:38.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-06T00:26:38.262 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-06T00:26:38.269 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-06T00:26:38.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-06T00:26:38.302 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-06T00:26:38.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-06T00:26:38.609 INFO:teuthology.orchestra.run.smithi078.stdout:47244640259 2024-05-06T00:26:38.626 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.3 2024-05-06T00:26:38.690 INFO:teuthology.orchestra.run.smithi078.stdout:47244640263 2024-05-06T00:26:38.692 INFO:teuthology.orchestra.run.smithi078.stdout:47244640262 2024-05-06T00:26:38.699 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-05-06T00:26:38.714 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640262 for osd.2 2024-05-06T00:26:38.714 DEBUG:teuthology.parallel:result is None 2024-05-06T00:26:38.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640261 for osd.0 2024-05-06T00:26:38.723 DEBUG:teuthology.parallel:result is None 2024-05-06T00:26:38.723 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640263 for osd.1 2024-05-06T00:26:38.723 DEBUG:teuthology.parallel:result is None 2024-05-06T00:26:38.778 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-05-06T00:26:38.801 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640261 for osd.4 2024-05-06T00:26:38.801 DEBUG:teuthology.parallel:result is None 2024-05-06T00:26:38.804 INFO:teuthology.orchestra.run.smithi078.stdout:47244640263 2024-05-06T00:26:38.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640263 for osd.5 2024-05-06T00:26:38.820 DEBUG:teuthology.parallel:result is None 2024-05-06T00:26:39.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-06T00:26:39.923 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-05-06T00:26:39.943 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640261 for osd.3 2024-05-06T00:26:39.943 DEBUG:teuthology.parallel:result is None 2024-05-06T00:26:39.943 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-06T00:26:39.944 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:26:40.222 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:40.223 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:26:40.238 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-05-06T00:26:38.395293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164264,"kb_used_data":2096,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066776,"statfs":{"total":579820584960,"available":579652378624,"internally_reserved":0,"allocated":2146304,"data_stored":1491662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001033"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637879+0000","last_change":"2024-05-06T00:26:36.150377+0000","last_active":"2024-05-06T00:26:36.637879+0000","last_peered":"2024-05-06T00:26:36.637879+0000","last_clean":"2024-05-06T00:26:36.637879+0000","last_became_active":"2024-05-06T00:26:36.149496+0000","last_became_peered":"2024-05-06T00:26:36.149496+0000","last_unstale":"2024-05-06T00:26:36.637879+0000","last_undegraded":"2024-05-06T00:26:36.637879+0000","last_fullsized":"2024-05-06T00:26:36.637879+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:54.410549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.739682+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:26:36.739682+0000","last_peered":"2024-05-06T00:26:36.739682+0000","last_clean":"2024-05-06T00:26:36.739682+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:26:36.739682+0000","last_undegraded":"2024-05-06T00:26:36.739682+0000","last_fullsized":"2024-05-06T00:26:36.739682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637874+0000","last_change":"2024-05-06T00:26:35.642448+0000","last_active":"2024-05-06T00:26:36.637874+0000","last_peered":"2024-05-06T00:26:36.637874+0000","last_clean":"2024-05-06T00:26:36.637874+0000","last_became_active":"2024-05-06T00:26:35.641678+0000","last_became_peered":"2024-05-06T00:26:35.641678+0000","last_unstale":"2024-05-06T00:26:36.637874+0000","last_undegraded":"2024-05-06T00:26:36.637874+0000","last_fullsized":"2024-05-06T00:26:36.637874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:32:32.988296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.739653+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:26:36.739653+0000","last_peered":"2024-05-06T00:26:36.739653+0000","last_clean":"2024-05-06T00:26:36.739653+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:26:36.739653+0000","last_undegraded":"2024-05-06T00:26:36.739653+0000","last_fullsized":"2024-05-06T00:26:36.739653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.742938+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:26:36.742938+0000","last_peered":"2024-05-06T00:26:36.742938+0000","last_clean":"2024-05-06T00:26:36.742938+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:26:36.742938+0000","last_undegraded":"2024-05-06T00:26:36.742938+0000","last_fullsized":"2024-05-06T00:26:36.742938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.634086+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:26:36.634086+0000","last_peered":"2024-05-06T00:26:36.634086+0000","last_clean":"2024-05-06T00:26:36.634086+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:26:36.634086+0000","last_undegraded":"2024-05-06T00:26:36.634086+0000","last_fullsized":"2024-05-06T00:26:36.634086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637883+0000","last_change":"2024-05-06T00:26:35.652664+0000","last_active":"2024-05-06T00:26:36.637883+0000","last_peered":"2024-05-06T00:26:36.637883+0000","last_clean":"2024-05-06T00:26:36.637883+0000","last_became_active":"2024-05-06T00:26:35.652017+0000","last_became_peered":"2024-05-06T00:26:35.652017+0000","last_unstale":"2024-05-06T00:26:36.637883+0000","last_undegraded":"2024-05-06T00:26:36.637883+0000","last_fullsized":"2024-05-06T00:26:36.637883+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:34.935271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.742919+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:26:36.742919+0000","last_peered":"2024-05-06T00:26:36.742919+0000","last_clean":"2024-05-06T00:26:36.742919+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:26:36.742919+0000","last_undegraded":"2024-05-06T00:26:36.742919+0000","last_fullsized":"2024-05-06T00:26:36.742919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":108,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637591+0000","last_change":"2024-05-06T00:26:30.239639+0000","last_active":"2024-05-06T00:26:36.637591+0000","last_peered":"2024-05-06T00:26:36.637591+0000","last_clean":"2024-05-06T00:26:36.637591+0000","last_became_active":"2024-05-06T00:26:30.237552+0000","last_became_peered":"2024-05-06T00:26:30.237552+0000","last_unstale":"2024-05-06T00:26:36.637591+0000","last_undegraded":"2024-05-06T00:26:36.637591+0000","last_fullsized":"2024-05-06T00:26:36.637591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:44:57.011856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":642189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":642189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:26:40.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:26:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:40.506 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:26:40.521 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-05-06T00:26:40.395875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164264,"kb_used_data":2096,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066776,"statfs":{"total":579820584960,"available":579652378624,"internally_reserved":0,"allocated":2146304,"data_stored":1491662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637879+0000","last_change":"2024-05-06T00:26:36.150377+0000","last_active":"2024-05-06T00:26:36.637879+0000","last_peered":"2024-05-06T00:26:36.637879+0000","last_clean":"2024-05-06T00:26:36.637879+0000","last_became_active":"2024-05-06T00:26:36.149496+0000","last_became_peered":"2024-05-06T00:26:36.149496+0000","last_unstale":"2024-05-06T00:26:36.637879+0000","last_undegraded":"2024-05-06T00:26:36.637879+0000","last_fullsized":"2024-05-06T00:26:36.637879+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:54.410549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.739682+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:26:36.739682+0000","last_peered":"2024-05-06T00:26:36.739682+0000","last_clean":"2024-05-06T00:26:36.739682+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:26:36.739682+0000","last_undegraded":"2024-05-06T00:26:36.739682+0000","last_fullsized":"2024-05-06T00:26:36.739682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637874+0000","last_change":"2024-05-06T00:26:35.642448+0000","last_active":"2024-05-06T00:26:36.637874+0000","last_peered":"2024-05-06T00:26:36.637874+0000","last_clean":"2024-05-06T00:26:36.637874+0000","last_became_active":"2024-05-06T00:26:35.641678+0000","last_became_peered":"2024-05-06T00:26:35.641678+0000","last_unstale":"2024-05-06T00:26:36.637874+0000","last_undegraded":"2024-05-06T00:26:36.637874+0000","last_fullsized":"2024-05-06T00:26:36.637874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:32:32.988296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.739653+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:26:36.739653+0000","last_peered":"2024-05-06T00:26:36.739653+0000","last_clean":"2024-05-06T00:26:36.739653+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:26:36.739653+0000","last_undegraded":"2024-05-06T00:26:36.739653+0000","last_fullsized":"2024-05-06T00:26:36.739653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.742938+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:26:36.742938+0000","last_peered":"2024-05-06T00:26:36.742938+0000","last_clean":"2024-05-06T00:26:36.742938+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:26:36.742938+0000","last_undegraded":"2024-05-06T00:26:36.742938+0000","last_fullsized":"2024-05-06T00:26:36.742938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.634086+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:26:36.634086+0000","last_peered":"2024-05-06T00:26:36.634086+0000","last_clean":"2024-05-06T00:26:36.634086+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:26:36.634086+0000","last_undegraded":"2024-05-06T00:26:36.634086+0000","last_fullsized":"2024-05-06T00:26:36.634086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637883+0000","last_change":"2024-05-06T00:26:35.652664+0000","last_active":"2024-05-06T00:26:36.637883+0000","last_peered":"2024-05-06T00:26:36.637883+0000","last_clean":"2024-05-06T00:26:36.637883+0000","last_became_active":"2024-05-06T00:26:35.652017+0000","last_became_peered":"2024-05-06T00:26:35.652017+0000","last_unstale":"2024-05-06T00:26:36.637883+0000","last_undegraded":"2024-05-06T00:26:36.637883+0000","last_fullsized":"2024-05-06T00:26:36.637883+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:34.935271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.742919+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:26:36.742919+0000","last_peered":"2024-05-06T00:26:36.742919+0000","last_clean":"2024-05-06T00:26:36.742919+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:26:36.742919+0000","last_undegraded":"2024-05-06T00:26:36.742919+0000","last_fullsized":"2024-05-06T00:26:36.742919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":108,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637591+0000","last_change":"2024-05-06T00:26:30.239639+0000","last_active":"2024-05-06T00:26:36.637591+0000","last_peered":"2024-05-06T00:26:36.637591+0000","last_clean":"2024-05-06T00:26:36.637591+0000","last_became_active":"2024-05-06T00:26:30.237552+0000","last_became_peered":"2024-05-06T00:26:30.237552+0000","last_unstale":"2024-05-06T00:26:36.637591+0000","last_undegraded":"2024-05-06T00:26:36.637591+0000","last_fullsized":"2024-05-06T00:26:36.637591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:44:57.011856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":642189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":642189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:26:40.522 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-06T00:26:40.522 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-06T00:26:40.523 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-06T00:26:40.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-06T00:26:40.828 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:40.828 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-06T00:26:40.844 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-06T00:26:40.845 INFO:teuthology.run_tasks:Running task thrashosds... 2024-05-06T00:26:40.856 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-05-06T00:26:40.856 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'b45ed4400464709677363de54673bcce970bcb27'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'b45ed4400464709677363de54673bcce970bcb27'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': 'ed0ddec0117930fb8b53c34226cd9519ed394ff9'}} 2024-05-06T00:26:40.857 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-05-06T00:26:40.857 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-05-06T00:26:41.177 INFO:tasks.thrashosds:Beginning thrashosds... 2024-05-06T00:26:41.178 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-06T00:26:41.178 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:26:41.458 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:41.459 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:26:41.475 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-05-06T00:26:40.395875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164264,"kb_used_data":2096,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066776,"statfs":{"total":579820584960,"available":579652378624,"internally_reserved":0,"allocated":2146304,"data_stored":1491662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637879+0000","last_change":"2024-05-06T00:26:36.150377+0000","last_active":"2024-05-06T00:26:36.637879+0000","last_peered":"2024-05-06T00:26:36.637879+0000","last_clean":"2024-05-06T00:26:36.637879+0000","last_became_active":"2024-05-06T00:26:36.149496+0000","last_became_peered":"2024-05-06T00:26:36.149496+0000","last_unstale":"2024-05-06T00:26:36.637879+0000","last_undegraded":"2024-05-06T00:26:36.637879+0000","last_fullsized":"2024-05-06T00:26:36.637879+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:54.410549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.739682+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:26:36.739682+0000","last_peered":"2024-05-06T00:26:36.739682+0000","last_clean":"2024-05-06T00:26:36.739682+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:26:36.739682+0000","last_undegraded":"2024-05-06T00:26:36.739682+0000","last_fullsized":"2024-05-06T00:26:36.739682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637874+0000","last_change":"2024-05-06T00:26:35.642448+0000","last_active":"2024-05-06T00:26:36.637874+0000","last_peered":"2024-05-06T00:26:36.637874+0000","last_clean":"2024-05-06T00:26:36.637874+0000","last_became_active":"2024-05-06T00:26:35.641678+0000","last_became_peered":"2024-05-06T00:26:35.641678+0000","last_unstale":"2024-05-06T00:26:36.637874+0000","last_undegraded":"2024-05-06T00:26:36.637874+0000","last_fullsized":"2024-05-06T00:26:36.637874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:32:32.988296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.739653+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:26:36.739653+0000","last_peered":"2024-05-06T00:26:36.739653+0000","last_clean":"2024-05-06T00:26:36.739653+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:26:36.739653+0000","last_undegraded":"2024-05-06T00:26:36.739653+0000","last_fullsized":"2024-05-06T00:26:36.739653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.742938+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:26:36.742938+0000","last_peered":"2024-05-06T00:26:36.742938+0000","last_clean":"2024-05-06T00:26:36.742938+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:26:36.742938+0000","last_undegraded":"2024-05-06T00:26:36.742938+0000","last_fullsized":"2024-05-06T00:26:36.742938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.634086+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:26:36.634086+0000","last_peered":"2024-05-06T00:26:36.634086+0000","last_clean":"2024-05-06T00:26:36.634086+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:26:36.634086+0000","last_undegraded":"2024-05-06T00:26:36.634086+0000","last_fullsized":"2024-05-06T00:26:36.634086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637883+0000","last_change":"2024-05-06T00:26:35.652664+0000","last_active":"2024-05-06T00:26:36.637883+0000","last_peered":"2024-05-06T00:26:36.637883+0000","last_clean":"2024-05-06T00:26:36.637883+0000","last_became_active":"2024-05-06T00:26:35.652017+0000","last_became_peered":"2024-05-06T00:26:35.652017+0000","last_unstale":"2024-05-06T00:26:36.637883+0000","last_undegraded":"2024-05-06T00:26:36.637883+0000","last_fullsized":"2024-05-06T00:26:36.637883+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:34.935271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.742919+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:26:36.742919+0000","last_peered":"2024-05-06T00:26:36.742919+0000","last_clean":"2024-05-06T00:26:36.742919+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:26:36.742919+0000","last_undegraded":"2024-05-06T00:26:36.742919+0000","last_fullsized":"2024-05-06T00:26:36.742919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":108,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637591+0000","last_change":"2024-05-06T00:26:30.239639+0000","last_active":"2024-05-06T00:26:36.637591+0000","last_peered":"2024-05-06T00:26:36.637591+0000","last_clean":"2024-05-06T00:26:36.637591+0000","last_became_active":"2024-05-06T00:26:30.237552+0000","last_became_peered":"2024-05-06T00:26:30.237552+0000","last_unstale":"2024-05-06T00:26:36.637591+0000","last_undegraded":"2024-05-06T00:26:36.637591+0000","last_fullsized":"2024-05-06T00:26:36.637591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:44:57.011856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":642189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":642189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:26:41.476 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:26:41.753 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:41.754 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:26:41.770 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-05-06T00:26:40.395875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164264,"kb_used_data":2096,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066776,"statfs":{"total":579820584960,"available":579652378624,"internally_reserved":0,"allocated":2146304,"data_stored":1491662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637879+0000","last_change":"2024-05-06T00:26:36.150377+0000","last_active":"2024-05-06T00:26:36.637879+0000","last_peered":"2024-05-06T00:26:36.637879+0000","last_clean":"2024-05-06T00:26:36.637879+0000","last_became_active":"2024-05-06T00:26:36.149496+0000","last_became_peered":"2024-05-06T00:26:36.149496+0000","last_unstale":"2024-05-06T00:26:36.637879+0000","last_undegraded":"2024-05-06T00:26:36.637879+0000","last_fullsized":"2024-05-06T00:26:36.637879+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:54.410549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.739682+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:26:36.739682+0000","last_peered":"2024-05-06T00:26:36.739682+0000","last_clean":"2024-05-06T00:26:36.739682+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:26:36.739682+0000","last_undegraded":"2024-05-06T00:26:36.739682+0000","last_fullsized":"2024-05-06T00:26:36.739682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637874+0000","last_change":"2024-05-06T00:26:35.642448+0000","last_active":"2024-05-06T00:26:36.637874+0000","last_peered":"2024-05-06T00:26:36.637874+0000","last_clean":"2024-05-06T00:26:36.637874+0000","last_became_active":"2024-05-06T00:26:35.641678+0000","last_became_peered":"2024-05-06T00:26:35.641678+0000","last_unstale":"2024-05-06T00:26:36.637874+0000","last_undegraded":"2024-05-06T00:26:36.637874+0000","last_fullsized":"2024-05-06T00:26:36.637874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:32:32.988296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.739653+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:26:36.739653+0000","last_peered":"2024-05-06T00:26:36.739653+0000","last_clean":"2024-05-06T00:26:36.739653+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:26:36.739653+0000","last_undegraded":"2024-05-06T00:26:36.739653+0000","last_fullsized":"2024-05-06T00:26:36.739653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.742938+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:26:36.742938+0000","last_peered":"2024-05-06T00:26:36.742938+0000","last_clean":"2024-05-06T00:26:36.742938+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:26:36.742938+0000","last_undegraded":"2024-05-06T00:26:36.742938+0000","last_fullsized":"2024-05-06T00:26:36.742938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.634086+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:26:36.634086+0000","last_peered":"2024-05-06T00:26:36.634086+0000","last_clean":"2024-05-06T00:26:36.634086+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:26:36.634086+0000","last_undegraded":"2024-05-06T00:26:36.634086+0000","last_fullsized":"2024-05-06T00:26:36.634086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637883+0000","last_change":"2024-05-06T00:26:35.652664+0000","last_active":"2024-05-06T00:26:36.637883+0000","last_peered":"2024-05-06T00:26:36.637883+0000","last_clean":"2024-05-06T00:26:36.637883+0000","last_became_active":"2024-05-06T00:26:35.652017+0000","last_became_peered":"2024-05-06T00:26:35.652017+0000","last_unstale":"2024-05-06T00:26:36.637883+0000","last_undegraded":"2024-05-06T00:26:36.637883+0000","last_fullsized":"2024-05-06T00:26:36.637883+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:34.935271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.742919+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:26:36.742919+0000","last_peered":"2024-05-06T00:26:36.742919+0000","last_clean":"2024-05-06T00:26:36.742919+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:26:36.742919+0000","last_undegraded":"2024-05-06T00:26:36.742919+0000","last_fullsized":"2024-05-06T00:26:36.742919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":108,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-06T00:26:36.637591+0000","last_change":"2024-05-06T00:26:30.239639+0000","last_active":"2024-05-06T00:26:36.637591+0000","last_peered":"2024-05-06T00:26:36.637591+0000","last_clean":"2024-05-06T00:26:36.637591+0000","last_became_active":"2024-05-06T00:26:30.237552+0000","last_became_peered":"2024-05-06T00:26:30.237552+0000","last_unstale":"2024-05-06T00:26:36.637591+0000","last_undegraded":"2024-05-06T00:26:36.637591+0000","last_fullsized":"2024-05-06T00:26:36.637591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:44:57.011856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":642189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":51821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":642189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:26:41.771 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-06T00:26:41.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-05-06T00:26:42.040 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 17 2024-05-06T00:26:42.040 INFO:teuthology.orchestra.run.smithi078.stdout:fsid b583c614-b2ca-4769-90c0-44726bf69b34 2024-05-06T00:26:42.040 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-05-06T00:26:16.923985+0000 2024-05-06T00:26:42.040 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-05-06T00:26:36.630161+0000 2024-05-06T00:26:42.040 INFO:teuthology.orchestra.run.smithi078.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-06T00:26:42.040 INFO:teuthology.orchestra.run.smithi078.stdout:crush_version 4 2024-05-06T00:26:42.040 INFO:teuthology.orchestra.run.smithi078.stdout:full_ratio 0.95 2024-05-06T00:26:42.040 INFO:teuthology.orchestra.run.smithi078.stdout:backfillfull_ratio 0.9 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:nearfull_ratio 0.85 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:require_min_compat_client luminous 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:min_compat_client jewel 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:require_osd_release squid 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:stretch_mode_enabled false 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.06 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 application rbd read_balance_score 2.25 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:max_osd 6 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6816/1427564096,v1:172.21.15.78:6817/1427564096] [v2:172.21.15.78:6818/1427564096,v1:172.21.15.78:6819/1427564096] exists,up d31b9a6a-864b-4c24-99db-8cc6cf3a5c24 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6801/2663808859,v1:172.21.15.78:6803/2663808859] [v2:172.21.15.78:6805/2663808859,v1:172.21.15.78:6807/2663808859] exists,up 86509826-0e53-4678-8260-17c97c05af54 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6800/385938386,v1:172.21.15.78:6802/385938386] [v2:172.21.15.78:6804/385938386,v1:172.21.15.78:6806/385938386] exists,up 8973b524-521e-41c0-87f2-85cb390fbfeb 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6811/4178038317,v1:172.21.15.146:6812/4178038317] [v2:172.21.15.146:6814/4178038317,v1:172.21.15.146:6816/4178038317] exists,up 3b0d07bd-132e-48ff-83ac-7182eee05df9 2024-05-06T00:26:42.041 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6808/1052555350,v1:172.21.15.146:6809/1052555350] [v2:172.21.15.146:6810/1052555350,v1:172.21.15.146:6813/1052555350] exists,up 0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2 2024-05-06T00:26:42.042 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6800/2019146845,v1:172.21.15.146:6801/2019146845] [v2:172.21.15.146:6802/2019146845,v1:172.21.15.146:6803/2019146845] exists,up 34b61f5f-e407-4414-8bc5-11263bbe261a 2024-05-06T00:26:42.057 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6816/1427564096,v1:172.21.15.78:6817/1427564096] [v2:172.21.15.78:6818/1427564096,v1:172.21.15.78:6819/1427564096] exists,up d31b9a6a-864b-4c24-99db-8cc6cf3a5c24', 'osd.1 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6801/2663808859,v1:172.21.15.78:6803/2663808859] [v2:172.21.15.78:6805/2663808859,v1:172.21.15.78:6807/2663808859] exists,up 86509826-0e53-4678-8260-17c97c05af54', 'osd.2 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6800/385938386,v1:172.21.15.78:6802/385938386] [v2:172.21.15.78:6804/385938386,v1:172.21.15.78:6806/385938386] exists,up 8973b524-521e-41c0-87f2-85cb390fbfeb', 'osd.3 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6811/4178038317,v1:172.21.15.146:6812/4178038317] [v2:172.21.15.146:6814/4178038317,v1:172.21.15.146:6816/4178038317] exists,up 3b0d07bd-132e-48ff-83ac-7182eee05df9', 'osd.4 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6808/1052555350,v1:172.21.15.146:6809/1052555350] [v2:172.21.15.146:6810/1052555350,v1:172.21.15.146:6813/1052555350] exists,up 0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2', 'osd.5 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6800/2019146845,v1:172.21.15.146:6801/2019146845] [v2:172.21.15.146:6802/2019146845,v1:172.21.15.146:6803/2019146845] exists,up 34b61f5f-e407-4414-8bc5-11263bbe261a'] 2024-05-06T00:26:42.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mon.a", 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster": "ceph", 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout: "admin_socket_mode": "", 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_client_required": "cephx, none", 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_cluster_required": "cephx", 2024-05-06T00:26:42.168 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_debug": "true", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_service_required": "cephx", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_service_ticket_ttl": "240.000000", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_supported": "cephx", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_aio": "true", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_aio_max_queue_depth": "1024", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_aio_poll_ms": "250", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_aio_reap_max": "16", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_async_discard_threads": "0", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_block_size": "4096", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_debug_aio": "false", 2024-05-06T00:26:42.169 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_debug_inflight_ios": "false", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_enable_discard": "false", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_flock_retry": "3", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_flock_retry_interval": "0.100000", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_inject_crash": "0", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_inject_crash_flush_delay": "2", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_ioring": "false", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_ioring_hipri": "false", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_ioring_sqthread_poll": "false", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_read_buffer_alignment": "4096", 2024-05-06T00:26:42.170 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_type": "", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_alloc_size": "1048576", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_allocator": "hybrid", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_buffered_io": "true", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_check_for_zeros": "false", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_check_volume_selector_often": "false", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_compact_log_sync": "false", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_log_compact_min_size": "16777216", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_log_replay_check_allocations": "true", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_max_log_runway": "4194304", 2024-05-06T00:26:42.171 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_max_prefetch": "1048576", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_min_flush_size": "524288", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_min_log_runway": "1048576", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_replay_recovery": "false", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_shared_alloc_size": "65536", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_sync_write": "false", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_allocation_from_file": "true", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_allocator": "hybrid", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-05-06T00:26:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blobid_prealloc": "10240", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_create": "true", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_db_create": "false", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_db_path": "", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_db_size": "0", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_path": "", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_preallocate_file": "false", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_size": "107374182400", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_wal_create": "false", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_wal_path": "", 2024-05-06T00:26:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_wal_size": "100663296", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bluefs": "true", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bluefs_env_mirror": "false", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_autotune": "true", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-05-06T00:26:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_size": "0", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_type": "2q", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_clone_cow": "true", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_algorithm": "snappy", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_max_blob_size": "0", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-05-06T00:26:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_min_blob_size": "0", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_mode": "none", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_csum_type": "crc32c", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_enforce_settings": "default", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_freelist": "false", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_fsck_abort": "false", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-05-06T00:26:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_inject_read_err": "false", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_legacy_omap": "false", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_prefill": "0.000000", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_small_allocations": "0", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_default_buffered_read": "true", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_default_buffered_write": "false", 2024-05-06T00:26:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_deferred_batch_ops": "0", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_elastic_shared_blobs": "true", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fail_eio": "false", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-05-06T00:26:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_mkfs": "true", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_mount": "false", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_umount": "false", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_ignore_data_csum": "false", 2024-05-06T00:26:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_kvbackend": "rocksdb", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_log_op_age": "5.000000", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_alloc_size": "0", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_blob_size": "0", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_defer_interval": "3.000000", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_deferred_txc": "32", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_min_alloc_size": "0", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-05-06T00:26:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_nid_prealloc": "1024", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_prefer_deferred_size": "0", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_qfsck_on_mount": "true", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_retry_disk_reads": "3", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_rocksdb_cf": "true", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_rocksdb_options_annex": "", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_spdk_coremask": "0x1", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_spdk_io_sleep": "5", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_spdk_max_io_completion": "0", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_spdk_mem": "512", 2024-05-06T00:26:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_sync_submit_transaction": "false", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_bytes": "67108864", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_cost_per_io": "0", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_tracing": "false", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_volume_selection_reserved": "0", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-05-06T00:26:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_zero_block_detection": "false", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_action_update_interval": "2", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_mount_timeout": "10", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-05-06T00:26:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_cluster_require_signatures": "false", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_cluster_require_version": "2", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_require_signatures": "false", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_require_version": "2", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_service_require_signatures": "false", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_service_require_version": "2", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_sign_messages": "true", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "chdir": "", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_acl_type": "", 2024-05-06T00:26:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_asio_thread_count": "2", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_cache_mid": "0.750000", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_cache_size": "16384", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_caps_release_delay": "5", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_check_pool_perm": "true", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_collect_and_send_global_metrics": "false", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_debug_force_sync_read": "false", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_debug_getattr_caps": "false", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_debug_inject_tick_delay": "0", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_die_on_failed_remount": "false", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_dirsize_rbytes": "true", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_force_lazyio": "false", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_fs": "", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_inject_fixed_oldest_tid": "false", 2024-05-06T00:26:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_inject_release_failure": "false", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_max_inline_size": "4096", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_max_retries_on_remount_failure": "5", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mds_namespace": "", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_metadata": "", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mount_gid": "-1", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mount_timeout": "300", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mount_uid": "-1", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mountpoint": "/", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_notify_timeout": "10", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc": "true", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_max_dirty": "104857600", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_max_dirty_age": "5.000000", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_max_objects": "1000", 2024-05-06T00:26:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_size": "209715200", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_target_dirty": "8388608", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_permissions": "true", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_quota": "true", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_quota_df": "true", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_readahead_max_bytes": "0", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_readahead_max_periods": "4", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_readahead_min": "131072", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_reconnect_stale": "false", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_shutdown_timeout": "30", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_snapdir": ".snap", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_tick_interval": "1", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_trace": "", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_try_dentry_invalidate": "false", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_use_faked_inos": "false", 2024-05-06T00:26:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_use_random_mds": "false", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_graylog": "false", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_graylog_port": "12201", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_monitors": "default=true", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_syslog": "false", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_syslog_level": "info", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "-", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_network": "", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_network_interface": "", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "compressor_zlib_isal": "false", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "compressor_zlib_level": "5", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "compressor_zlib_winsize": "-15", 2024-05-06T00:26:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "compressor_zstd_level": "1", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_alien_op_num_threads": "6", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_alien_thread_cpu_cores": "", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_osd_obc_lru_size": "512", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_osd_stat_interval": "0", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_seastar_cpu_cores": "", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location_hook": "", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location_hook_timeout": "10", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "daemonize": "false", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "daos_pool": "tank", 2024-05-06T00:26:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "dbstore_db_name_prefix": "dbstore", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_alienstore": "0/5", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_allow_any_pool_priority": "false", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_asok": "1/5", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_asok_assert_abort": "false", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_asserts_on_shutdown": "true", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_auth": "20/20", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_bdev": "1/3", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_bluefs": "1/5", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_bluestore": "1/5", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_buffer": "0/1", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_ceph_dedup": "0/5", 2024-05-06T00:26:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_ceph_exporter": "1/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_cephfs_mirror": "0/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_cephsqlite": "0/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_client": "0/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_compressor": "1/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_context": "0/1", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_crimson_interrupt": "0/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_crush": "1/1", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_crypto": "1/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_cyanstore": "0/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_deliberately_leak_memory": "false", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_disable_randomized_ping": "false", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_dpdk": "1/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_eventtrace": "1/5", 2024-05-06T00:26:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_filer": "0/1", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_filestore": "1/3", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_finisher": "1/1", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_fuse": "1/5", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_heartbeat_testing_span": "0", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_heartbeatmap": "1/5", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_immutable_obj_cache": "0/5", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_javaclient": "1/5", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_journal": "1/3", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_journaler": "0/5", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_kstore": "1/5", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_lockdep": "0/1", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mclock": "1/5", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds": "1/5", 2024-05-06T00:26:42.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_balancer": "1/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_locker": "1/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_log": "1/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_log_expire": "1/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_migrator": "1/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_quiesce": "3/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_memstore": "1/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mgr": "2/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mgrc": "1/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mon": "20/20", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_monc": "0/10", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_ms": "1/1", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_none": "0/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_objclass": "0/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_objectcacher": "0/5", 2024-05-06T00:26:42.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_objecter": "0/1", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_optracker": "0/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_osd": "1/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_paxos": "20/20", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_perfcounter": "1/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_prioritycache": "1/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rados": "0/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rbd": "0/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rbd_mirror": "0/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rbd_pwl": "0/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rbd_replay": "0/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_refs": "0/0", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_reserver": "1/1", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw": "1/5", 2024-05-06T00:26:42.194 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_access": "1/5", 2024-05-06T00:26:42.198 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_datacache": "1/5", 2024-05-06T00:26:42.198 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_dbstore": "1/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_flight": "1/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_lifecycle": "1/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_notification": "1/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_sync": "1/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rocksdb": "4/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_backref": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_cache": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_cleaner": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_device": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_epm": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_journal": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_lba": "0/5", 2024-05-06T00:26:42.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_odata": "0/5", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_omap": "0/5", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_onode": "0/5", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_t": "0/5", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_tm": "0/5", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_striper": "0/1", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_test": "0/5", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_throttle": "1/1", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_timer": "0/1", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_tp": "0/5", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_trace": "1/5", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "device_failure_prediction_mode": "none", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-05-06T00:26:42.200 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "err_to_graylog": "false", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "err_to_journald": "false", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "err_to_stderr": "true", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "err_to_syslog": "false", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event_tracing": "false", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_addr": "0.0.0.0", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_http_port": "9926", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_prio_limit": "5", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_sort_metrics": "true", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_stats_period": "5", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "fake_statfs_for_testing": "0", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "fatal_signal_handlers": "true", 2024-05-06T00:26:42.201 INFO:teuthology.orchestra.run.smithi078.stdout: "filer_max_purge_ops": "10", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filer_max_truncate_ops": "128", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_apply_finisher_threads": "1", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_blackhole": "false", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_btrfs_clone_range": "true", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_btrfs_snap": "true", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_caller_concurrency": "10", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_collect_device_partition_information": "true", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_commit_timeout": "600.000000", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_debug_inject_read_err": "false", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_debug_omap_check": "false", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_debug_verify_split": "false", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_dump_file": "", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-05-06T00:26:42.202 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fadvise": "true", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fail_eio": "true", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fd_cache_shards": "16", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fd_cache_size": "128", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fiemap": "false", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fiemap_threshold": "4096", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_index_retry_probability": "0.000000", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_inject_stall": "0", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_journal_parallel": "false", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_journal_trailing": "false", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_journal_writeahead": "false", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_kill_at": "0", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-05-06T00:26:42.203 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattr_size": "0", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattrs": "0", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattrs_other": "2", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_sync_interval": "5.000000", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_xattr_value_size": "0", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_merge_threshold": "-10", 2024-05-06T00:26:42.204 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_min_sync_interval": "0.010000", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_odsync_write": "false", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_omap_backend": "rocksdb", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_omap_backend_path": "", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_omap_header_cache_size": "1024", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_ondisk_finisher_threads": "1", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_op_thread_timeout": "60", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_op_threads": "2", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_punch_hole": "false", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-05-06T00:26:42.205 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_bytes": "104857600", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_ops": "50", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_seek_data_hole": "false", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_sloppy_crc": "false", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_splice": "false", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_split_multiple": "2", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_split_rand_factor": "20", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_update_to": "1000", 2024-05-06T00:26:42.206 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_enable": "true", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_xfs_extsize": "false", 2024-05-06T00:26:42.207 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_zfs_snap": "false", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fio_dir": "/tmp/fio", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b583c614-b2ca-4769-90c0-44726bf69b34", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_allow_other": "true", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_atomic_o_trunc": "true", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_big_writes": "true", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_debug": "false", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_default_permissions": "false", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_disable_pagecache": "false", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_max_write": "0", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_multithreaded": "true", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_require_active_mds": "true", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_set_user_groups": "true", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_splice_move": "true", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_splice_read": "true", 2024-05-06T00:26:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_splice_write": "true", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_syncfs_on_mksnap": "true", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_use_invalidate_cb": "true", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "gss_target_name": "ceph", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_file": "", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_inject_failure": "0", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_interval": "5", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "host": "smithi078", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_max_size": "1073741824", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-05-06T00:26:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_watermark": "0.900000", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "inject_early_sigterm": "false", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "jaeger_agent_port": "6799", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "jaeger_tracing_enable": "false", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_aio": "true", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_align_min_size": "65536", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_block_align": "true", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_block_size": "4096", 2024-05-06T00:26:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_dio": "true", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_discard": "false", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_force_aio": "false", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_ignore_corruption": "false", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_max_write_bytes": "10485760", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_max_write_entries": "100", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_replay_from": "0", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_throttle_high_multiple": "0.000000", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_throttle_max_multiple": "0.000000", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_write_header_frequency": "0", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_zero_on_create": "false", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journaler_prefetch_periods": "10", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journaler_prezero_periods": "5", 2024-05-06T00:26:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "journaler_write_head_interval": "15", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "keyfile": "", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_backend": "rocksdb", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_default_stripe_size": "65536", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_fsck_on_mount": "false", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_fsck_on_mount_deep": "true", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_max_bytes": "67108864", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_max_ops": "512", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_nid_prealloc": "1024", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_onode_map_size": "1024", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_sync_submit_transaction": "false", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_sync_transaction": "false", 2024-05-06T00:26:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "librados_thread_count": "2", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "lockdep": "false", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "lockdep_force_backtrace": "false", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_coarse_timestamps": "true", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_flush_on_exit": "false", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_graylog_host": "127.0.0.1", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_graylog_port": "12201", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_max_new": "1000", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_max_recent": "10000", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_stderr_prefix": "", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_stop_at_utilization": "0.970000", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": "true", 2024-05-06T00:26:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_graylog": "false", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_journald": "false", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_stderr": "false", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_syslog": "false", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "max_rotating_auth_attempts": "10", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_action_on_write_error": "1", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_alternate_name_max": "8192", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_asio_thread_count": "2", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_export_pin": "true", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_fragment_dirs": "true", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_fragment_interval": "5", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_fragment_size_max": "100000", 2024-05-06T00:26:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_idle_threshold": "0.000000", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_interval": "10", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_max": "-1", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_max_until": "-1", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_merge_size": "50", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_midchunk": "0.300000", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_min_rebalance": "0.100000", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_min_start": "0.200000", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_minchunk": "0.001000", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_mode": "0", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_need_max": "1.200000", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_need_min": "0.800000", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_overload_epochs": "2", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-05-06T00:26:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_sample_interval": "3.000000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_split_bits": "3", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_split_rd": "25000.000000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_split_size": "10000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_split_wr": "10000.000000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_target_decay": "10.000000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_beacon_grace": "15.000000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_beacon_interval": "4.000000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_beacon_mon_down_grace": "60", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_memory_limit": "4294967296", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_mid": "0.700000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_delay": "0", 2024-05-06T00:26:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_sleep": "200", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_splitauth": "true", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_threshold": "524288", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_release_free_interval": "10", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_reservation": "0.050000", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_trim_interval": "1", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_trim_threshold": "262144", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_client_delegate_inos_pct": "50", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_client_prealloc_inos": "1000", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_connect_bootstrapping": "false", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_damage_table_max_entries": "10000", 2024-05-06T00:26:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_debug_auth_pins": "false", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_debug_frag": "false", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_debug_scatterstat": "false", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_debug_subtrees": "false", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_decay_halflife": "5.000000", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_default_dir_hash": "2", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_defer_session_stale": "true", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_deny_all_reconnect": "false", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dir_keys_per_op": "16384", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dir_max_commit_size": "10", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dir_max_entries": "0", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dir_prefetch": "true", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dirstat_min_interval": "1.000000", 2024-05-06T00:26:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dump_cache_after_rejoin": "false", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dump_cache_on_map": "false", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dump_cache_threshold_file": "0", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_early_reply": "true", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_enable_op_tracker": "true", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_enforce_unique_name": "true", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_export_ephemeral_distributed": "true", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_export_ephemeral_random": "true", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_forward_all_requests_to_auth": "false", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-05-06T00:26:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_health_cache_threshold": "1.500000", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_health_summarize_threshold": "10", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_heartbeat_grace": "15.000000", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_heartbeat_reset_grace": "1000", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_health_dummy": "false", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_migrator_session_race": "false", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_join_fs": "", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_journal_format": "1", 2024-05-06T00:26:42.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_create_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_export_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_import_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_journal_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_journal_expire_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_journal_replay_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_link_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_mdstable_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_openc_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_rename_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_shutdown_at": "0", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_event_large_threshold": "524288", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_events_per_segment": "1024", 2024-05-06T00:26:42.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_major_segment_event_ratio": "12", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_max_events": "-1", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_max_segments": "128", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_pause": "false", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_segment_size": "0", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_skip_corrupt_events": "false", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_skip_unbounded_events": "false", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_trim_threshold": "128", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_warn_factor": "2.000000", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_caps_per_client": "1048576", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_completed_flushes": "100000", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_completed_requests": "100000", 2024-05-06T00:26:42.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_export_size": "20971520", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_file_recover": "32", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_purge_files": "64", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_purge_ops": "8192", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_snaps_per_dir": "100", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_metrics_update_interval": "2", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_min_caps_per_client": "100", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_min_caps_working_set": "10000", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_numa_node": "-1", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_complaint_time": "30.000000", 2024-05-06T00:26:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_history_duration": "600", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_history_size": "20", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_history_slow_op_size": "20", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_history_slow_op_threshold": "10", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_log_threshold": "5", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_ping_grace": "15", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_ping_interval": "5", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_max_caps": "30000", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_max_decay_threshold": "131072", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_warning_threshold": "262144", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_reconnect_timeout": "45.000000", 2024-05-06T00:26:42.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_replay_interval": "1.000000", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_root_ino_gid": "0", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_root_ino_uid": "0", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_blocklist_on_evict": "true", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_blocklist_on_timeout": "true", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-05-06T00:26:42.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_metadata_threshold": "16777216", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_shutdown_check": "0", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_skip_ino": "0", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_sleep_rank_change": "0.000000", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_snap_max_uid": "4294967294", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_snap_min_uid": "0", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_snap_rstat": "false", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_standby_replay_damaged": "false", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_symlink_recovery": "true", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_task_status_update_interval": "2.000000", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_thrash_exports": "0", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_thrash_fragments": "0", 2024-05-06T00:26:42.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_tick_interval": "5.000000", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_valgrind_exit": "false", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_verify_backtrace": "1", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_verify_scatter": "false", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_wipe_ino_prealloc": "false", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_wipe_sessions": "false", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool_debug": "false", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "memstore_debug_omit_block_device_write": "false", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "memstore_device_bytes": "1073741824", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "memstore_page_set": "false", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "memstore_page_size": "65536", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_client_bytes": "134217728", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_client_messages": "512", 2024-05-06T00:26:42.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_connect_retry_interval": "1.000000", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_disabled_modules": "", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_enable_op_tracker": "true", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_max_pg_creating": "1024", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_max_pg_num_change": "128", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_mds_bytes": "134217728", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_mds_messages": "128", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_mon_bytes": "134217728", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_mon_messages": "128", 2024-05-06T00:26:42.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_num_op_tracker_shard": "32", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_op_complaint_time": "30.000000", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_op_history_duration": "600", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_op_history_size": "20", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_op_history_slow_op_size": "20", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_op_log_threshold": "5", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_osd_bytes": "536870912", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_osd_messages": "8192", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_pool": "true", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_service_beacon_grace": "60.000000", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_standby_modules": "true", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_stats_period": "5", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_stats_threshold": "5", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_tick_period": "2", 2024-05-06T00:26:42.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_accept_timeout_factor": "2.000000", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_allow_pool_delete": "true", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_allow_pool_size_one": "true", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_auth_validate_all_caps": "true", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_bytes": "104857600", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_directed_command_retry": "5", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_interval": "3.000000", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_parallel": "3", 2024-05-06T00:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_log_interval": "1.000000", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_ping_interval": "10.000000", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_ping_timeout": "30.000000", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_target_rank": "-1", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_clock_drift_allowed": "1.000000", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_level": "debug", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_file": "true", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_graylog": "false", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_journald": "false", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_stderr": "false", 2024-05-06T00:26:42.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_compact_on_bootstrap": "false", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_compact_on_start": "false", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_compact_on_trim": "true", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_con_tracker_persist_interval": "10", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_con_tracker_score_halflife": "43200", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_config_key_max_entry_size": "65536", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cpu_threads": "4", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_crush_min_required_version": "hammer", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_daemon_bytes": "419430400", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_data_avail_crit": "5", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_data_avail_warn": "5", 2024-05-06T00:26:42.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_data_size_warn": "16106127360", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_block_osdmap_trim": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_dump_json": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_dump_transactions": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_extra_checks": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_no_require_reef": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_no_require_squid": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_delta_reset_interval": "10.000000", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_dns_srv_name": "ceph-mon", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_down_mkfs_grace": "120", 2024-05-06T00:26:42.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_down_uptime_grace": "60", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_election_default_strategy": "1", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_election_timeout": "5.000000", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_elector_ping_divisor": "2", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_elector_ping_timeout": "2.000000", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_enable_op_tracker": "true", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_fake_pool_delete": "false", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_force_quorum_join": "false", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_globalid_prealloc": "10000", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_detail_to_clog": "false", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_log_update_period": "5", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_max_detail": "50", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_to_clog": "true", 2024-05-06T00:26:42.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_to_clog_interval": "600", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_host": "172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.146", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_host_override": "", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_initial_members": "", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_keyvaluedb": "rocksdb", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_lease": "5.000000", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_log_full_interval": "50", 2024-05-06T00:26:42.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_log_max": "10000", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_log_max_summary": "50", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_log_entries_per_event": "4096", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_log_epochs": "500", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_mdsmap_epochs": "500", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_mgrmap_epochs": "500", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_osd": "10000", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_pg_per_osd": "10000", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_pool_pg_num": "65536", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-05-06T00:26:42.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mds_force_trim_to": "0", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mds_skip_sanity": "false", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_memory_autotune": "true", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_memory_target": "2147483648", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_beacon_grace": "30", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_digest_period": "5", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_inactive_grace": "60", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_mkfs_grace": "240", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_min_osdmap_epochs": "500", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_complaint_time": "30", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_history_duration": "600", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_history_size": "20", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_history_slow_op_size": "20", 2024-05-06T00:26:42.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_history_slow_op_threshold": "10", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_log_threshold": "5", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_auto_mark_in": "false", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_auto_mark_new_in": "true", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_cache_size": "500", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_cache_size_min": "134217728", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_crush_smoke_test": "true", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_destroyed_out_interval": "600", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_down_out_interval": "600", 2024-05-06T00:26:42.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_force_trim_to": "0", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_full_ratio": "0.950000", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_laggy_halflife": "3600", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_laggy_max_interval": "300", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_laggy_weight": "0.300000", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_max_initial_pgs": "1024", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_min_down_reporters": "2", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_prime_pg_temp": "true", 2024-05-06T00:26:42.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_report_timeout": "900", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_warn_num_repaired": "10", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_warn_op_age": "32.000000", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osdmap_full_prune_interval": "10", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osdmap_full_prune_min": "10000", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_stuck_threshold": "60", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-05-06T00:26:42.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_warn_min_objects": "10000", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_warn_min_per_osd": "0", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pool_quota_crit_threshold": "0", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pool_quota_warn_threshold": "0", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_probe_timeout": "2.000000", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_reweight_max_change": "0.050000", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_reweight_max_osds": "4", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_interval": "86400", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_max_keys": "100", 2024-05-06T00:26:42.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_timeout": "300", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_session_timeout": "300", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_smart_report_timeout": "5", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stat_smooth_intervals": "6", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stretch_pool_min_size": "2", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stretch_pool_size": "4", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_subscribe_interval": "86400.000000", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_debug": "false", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_max_payload_keys": "2000", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_max_payload_size": "1048576", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_provider_kill_at": "0", 2024-05-06T00:26:42.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_requester_kill_at": "0", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_timeout": "60.000000", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_target_pg_per_osd": "100", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_tick_interval": "5", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_timecheck_interval": "300.000000", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_use_min_delay_socket": "false", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_older_version_delay": "604800", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_filestore_osds": "false", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-05-06T00:26:42.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_misplaced": "false", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_older_version": "true", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_pool_no_app": "false", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_too_few_osds": "false", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": "", 2024-05-06T00:26:42.244 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_tracing_enabled": "false", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_op_threads": "3", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_buffer_size": "131072", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_cm": "false", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_device_name": "", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_dscp": "96", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_gid_idx": "0", 2024-05-06T00:26:42.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_local_gid": "", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_polling_us": "1000", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_port_num": "1", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_roce_ver": "1", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_send_buffers": "1024", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_sl": "3", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_support_srq": "true", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_type": "ib", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_reap_threshold": "5", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_before_connect": "false", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_ipv4": "true", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_ipv6": "false", 2024-05-06T00:26:42.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_msgr1": "true", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_msgr2": "true", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_port_max": "7568", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_port_min": "6800", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_prefer_ipv4": "false", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_retry_count": "3", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_retry_delay": "5", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_client": "false", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_mds": "false", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_mgr": "false", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_mon": "false", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_osd": "false", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_client_mode": "crc secure", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_cluster_mode": "crc secure", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_cluster_type": "", 2024-05-06T00:26:42.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_compress_secure": "false", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_connection_idle_timeout": "900", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_connection_ready_timeout": "10", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_crc_data": "true", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_crc_header": "true", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_bad_msg": "false", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_bug": "true", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_old_message": "true", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_skipped_message": "false", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_unhandled_msg": "false", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_coremask": "0xF", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_devs_allowlist": "", 2024-05-06T00:26:42.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_enable_tso": "true", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_hugepages": "", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_hw_flow_control": "true", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_lro": "true", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_memory_channel": "4", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_pmd": "", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_port_id": "0", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dump_corrupt_message_level": "1", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dump_on_send": "false", 2024-05-06T00:26:42.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_initial_backoff": "0.200000", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_delay_max": "1.000000", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_delay_probability": "0.000000", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_delay_type": "", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_internal_delays": "0.000000", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_network_congestion": "0", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_socket_failures": "5000", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_learn_addr_from_peer": "true", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_max_accept_failures": "4", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_max_backoff": "15.000000", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_mon_client_mode": "secure crc", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_mon_cluster_mode": "secure crc", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_mon_service_mode": "secure crc", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_osd_compress_min_size": "1024", 2024-05-06T00:26:42.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_osd_compress_mode": "none", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_osd_compression_algorithm": "snappy", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_pq_min_cost": "65536", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_public_type": "", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_service_mode": "crc secure", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_tcp_listen_backlog": "512", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_tcp_nodelay": "true", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_tcp_rcvbuf": "0", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_type": "async+posix", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "no_config_file": "false", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_completion_locks_per_session": "32", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_debug_inject_relock_delay": "false", 2024-05-06T00:26:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_inflight_op_bytes": "104857600", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_inflight_ops": "1024", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_inject_no_watch_ping": "false", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_tick_interval": "5.000000", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_timeout": "10.000000", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "objectstore_blackhole": "false", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "openssl_engine_opts": "", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_delay_time": "5.000000", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_hist_halflife": "1000", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_max_low_ops": "2", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_max_ops": "4", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_quantize_effort": "0.100000", 2024-05-06T00:26:42.252 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_slop": "0.020000", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_allow_recovery_below_min_size": "true", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_asio_thread_count": "2", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_async_recovery_min_cost": "100", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_auto_mark_unfound_lost": "false", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backfill_retry_interval": "30.000000", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backfill_scan_max": "512", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backfill_scan_min": "64", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backoff_on_degraded": "false", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backoff_on_peering": "false", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backoff_on_unfound": "true", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_beacon_report_interval": "300", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bench_duration": "30", 2024-05-06T00:26:42.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bench_max_block_size": "67108864", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bench_small_size_max_iops": "100", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_blkin_trace_all": "false", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_blocked_scrub_grace_period": "120", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_check_for_log_corruption": "false", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_class_update_on_start": "true", 2024-05-06T00:26:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_client_message_cap": "256", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_client_message_size_cap": "524288000", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_client_op_priority": "63", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_client_watch_timeout": "30", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_command_max_records": "256", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_compact_on_start": "false", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_crush_chooseleaf_type": "0", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_crush_initial_weight": "-1.000000", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_crush_update_on_start": "true", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_crush_update_weight_set": "true", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-05-06T00:26:42.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_drop_ping_duration": "0", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_feed_pullee": "-1", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_misdirected_ops": "false", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_no_acting_change": "false", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_no_purge_strays": "false", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_op_order": "false", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_pg_log_writeout": "false", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_pretend_recovery_active": "false", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_shutdown": "false", 2024-05-06T00:26:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_trim_objects": "false", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_verify_cached_snaps": "false", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_verify_missing_on_start": "false", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_verify_snaps": "false", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_keys": "1024", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_stride": "524288", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-05-06T00:26:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_default_data_pool_replay_window": "5", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_default_notify_timeout": "30", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_delete_sleep": "0.000000", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_discard_disconnected_ops": "true", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_enable_op_tracker": "true", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_extblkdev_plugins": "vdo", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_info": "true", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_shutdown": "true", 2024-05-06T00:26:42.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_shutdown_timeout": "15", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_function_tracing": "false", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_grace": "20", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_interval": "6", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_min_peers": "10", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_min_size": "2000", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_stale": "600", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_hit_set_max_size": "100000", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_hit_set_min_size": "1000", 2024-05-06T00:26:42.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_journal_flush_on_shutdown": "true", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_journal_size": "5120", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_kill_backfill_at": "0", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_loop_before_reset_tphandle": "64", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_cache_size": "50", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_dedup": "true", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_message_max": "40", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_message_max_bytes": "10485760", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_share_max_epochs": "40", 2024-05-06T00:26:42.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_attr_name_len": "100", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_attr_size": "0", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_backfills": "1", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_markdown_count": "5", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_markdown_period": "600", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_object_name_len": "2048", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_object_namespace_len": "256", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_object_size": "134217728", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_omap_entries_per_request": "1024", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_pg_blocked_by": "16", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_pg_log_entries": "10000", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_pgls": "1024", 2024-05-06T00:26:42.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_push_cost": "8388608", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_push_objects": "10", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_scrubs": "3", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_trimming_pgs": "2", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_write_op_reply_len": "64", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_write_size": "90", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_override_recovery_settings": "false", 2024-05-06T00:26:42.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_profile": "balanced", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_skip_benchmark": "false", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_base": "805306368", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_cache_min": "134217728", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-05-06T00:26:42.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_target": "4294967296", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_target_autotune": "false", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_min_pg_log_entries": "250", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_min_recovery_priority": "0", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_heartbeat_interval": "30", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_report_interval": "5", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_report_max_in_flight": "2", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_num_cache_shards": "32", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_num_op_tracker_shard": "32", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_numa_auto_affinity": "true", 2024-05-06T00:26:42.264 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_numa_node": "-1", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_numa_prefer_iface": "true", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_objecter_finishers": "1", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_objectstore": "bluestore", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_objectstore_fuse": "false", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_objectstore_tracing": "false", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_complaint_time": "30.000000", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_history_duration": "600", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_history_size": "20", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_history_slow_op_size": "20", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_log_threshold": "5", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_shards": "0", 2024-05-06T00:26:42.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_shards_hdd": "5", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_shards_ssd": "8", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_threads_per_shard": "0", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_pq_min_cost": "65536", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_queue": "mclock_scheduler", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_queue_cut_off": "high", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_thread_suicide_timeout": "150", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_thread_timeout": "15", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_open_classes_on_start": "true", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_os_flags": "0", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_peering_op_priority": "255", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_delete_cost": "1048576", 2024-05-06T00:26:42.266 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_delete_priority": "5", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_log_dups_tracked": "3000", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_log_trim_max": "10000", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_log_trim_min": "100", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_object_context_cache_count": "64", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-05-06T00:26:42.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_crimson": "false", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_crush_rule": "-1", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_ec_fast_read": "false", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_bulk": "false", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_nodelete": "false", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flags": "0", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_min_size": "0", 2024-05-06T00:26:42.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_pg_num": "32", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_pgp_num": "0", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_read_ratio": "70", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_size": "2", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_type": "replicated", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_use_gmt_hitset": "true", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_push_per_object_cost": "1000", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_read_ec_check_for_errors": "false", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recover_clone_overlap": "true", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recover_clone_overlap_limit": "10", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_cost": "20971520", 2024-05-06T00:26:42.269 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_delay_start": "0.000000", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_active": "0", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_active_hdd": "3", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_active_ssd": "10", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_chunk": "8388608", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_single_start": "1", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_op_priority": "3", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_op_warn_multiple": "16", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_priority": "5", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_retry_interval": "30.000000", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_sleep": "0.000000", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-05-06T00:26:42.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_repair_during_recovery": "false", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_requested_scrub_priority": "120", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_rollback_to_cluster_snap": "", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_auto_repair": "false", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_begin_hour": "0", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_begin_week_day": "0", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_chunk_max": "25", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_chunk_min": "5", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_cost": "52428800", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-05-06T00:26:42.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_during_recovery": "false", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_end_hour": "0", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_end_week_day": "0", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_event_cost": "4096", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_invalid_stats": "true", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_load_threshold": "0.500000", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_max_interval": "604800.000000", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_max_preemptions": "5", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_min_interval": "86400.000000", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_priority": "5", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_reservation_timeout": "300000", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_sleep": "0.000000", 2024-05-06T00:26:42.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_shutdown_pgref_assert": "false", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_skip_data_digest": "false", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_smart_report_timeout": "5", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_cost": "1048576", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_priority": "5", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_sleep": "0.000000", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_stats_update_period_scrubbing": "15", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-05-06T00:26:42.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_target_transaction_size": "30", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_mode": "writeback", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tracing": "false", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_use_stale_snap": "false", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-05-06T00:26:42.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osdc_blkin_trace_all": "false", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_kill_at": "0", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_max_join_drift": "10", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_min": "500", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_min_wait": "0.050000", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_propose_interval": "1.000000", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_service_trim_max": "500", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_service_trim_max_multiplier": "20", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_service_trim_min": "250", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_stash_full_interval": "25", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_trim_max": "500", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_trim_min": "250", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "perf": "true", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-05-06T00:26:42.275 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin_dir": "/usr/lib/ceph", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "-", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrv": "", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "public_bind_addr": "-", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "public_network": "", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "public_network_interface": "", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "qat_compressor_enabled": "false", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "qat_compressor_session_max_number": "256", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rados_mon_op_timeout": "0", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rados_osd_op_timeout": "0", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rados_tracing": "false", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_atime_update_interval": "60", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_balance_parent_reads": "false", 2024-05-06T00:26:42.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_balance_snap_reads": "false", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_blkin_trace_all": "false", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_blocklist_expire_seconds": "0", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_blocklist_on_break_lock": "true", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache": "true", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_block_writes_upfront": "false", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_max_dirty": "25165824", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_max_dirty_object": "0", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_policy": "writearound", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_size": "33554432", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_target_dirty": "16777216", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_clone_copy_on_read": "false", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_compression_hint": "none", 2024-05-06T00:26:42.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_concurrent_management_ops": "10", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_clone_format": "auto", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_data_pool": "", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_features": "61", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_format": "2", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_map_options": "", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_order": "22", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_pool": "rbd", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_stripe_count": "0", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_stripe_unit": "0", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_disable_zero_copy_writes": "true", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_discard_granularity_bytes": "65536", 2024-05-06T00:26:42.278 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_enable_alloc_hint": "true", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_io_scheduler": "simple", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_commit_age": "5.000000", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_flush_interval": "0", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_order": "24", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_pool": "", 2024-05-06T00:26:42.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_splay_width": "4", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_localize_parent_reads": "false", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_localize_snap_reads": "false", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_policy_type": "simple", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-05-06T00:26:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_autotune": "true", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_base": "805306368", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_target": "4294967296", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirroring_delete_delay": "0", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-05-06T00:26:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirroring_replay_delay": "0", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_move_to_trash_on_remove": "false", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mtime_update_interval": "60", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_non_blocking_aio": "true", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_op_thread_timeout": "60", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_op_threads": "1", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_parent_cache_enabled": "false", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_persistent_cache_mode": "disabled", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_persistent_cache_path": "/tmp", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_persistent_cache_size": "1073741824", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_plugins": "", 2024-05-06T00:26:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_bps_burst": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_bps_limit": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_exclude_ops": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_iops_burst": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_iops_limit": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_bps_burst": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_bps_limit": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_iops_burst": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_iops_limit": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_schedule_tick_min": "50", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_bps_burst": "0", 2024-05-06T00:26:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_bps_limit": "0", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_iops_burst": "0", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_iops_limit": "0", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_quiesce_notification_attempts": "10", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_read_from_replica_policy": "default", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_readahead_max_bytes": "524288", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_readahead_trigger_requests": "10", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_request_timed_out_seconds": "30", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_skip_partial_discard": "true", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_tracing": "false", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_validate_names": "true", 2024-05-06T00:26:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_validate_pool": "true", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "restapi_base_url": "", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "restapi_log_level": "", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_account_default_quota_max_size": "-1", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_acl_grants_max_num": "100", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_admin_entry": "admin", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_backend_store": "rados", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_barbican_url": "", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_beast_enable_async": "true", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_counters_cache": "false", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-05-06T00:26:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_index_max_aio": "128", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_quota_ttl": "600", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cache_enabled": "true", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cache_expiry_interval": "900", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cache_lru_size": "25000", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_config_store": "rados", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_content_length_compat": "false", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_copy_obj_progress": "true", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cors_rules_max_num": "100", 2024-05-06T00:26:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cross_domain_policy": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_default_encryption_key": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_addr": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_ca_path": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_client_cert": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_client_key": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_password": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_username": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_require_ssl": "true", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-05-06T00:26:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_suppress_logs": "true", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_addr": "", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_auth": "token", 2024-05-06T00:26:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_namespace": "", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_prefix": "", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_token_file": "", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_buffersize": "524288", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_low_speed_limit": "1024", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_low_speed_time": "300", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_tcp_keepalive": "0", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-05-06T00:26:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_fadvise": "4", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_host": "127.0.0.1", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_l1_fadvise": "4", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_port": "6379", 2024-05-06T00:26:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_log_changes_size": "1000", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_log_num_shards": "128", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_log_window": "30", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_notify_interval_msec": "0", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_sync_poll_interval": "20", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_sync_spawn_window": "20", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_data_log_backing": "fifo", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_region_info_oid": "default.region", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-05-06T00:26:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_defer_to_bucket_acls": "", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_data_res": "500.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-05-06T00:26:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dns_name": "", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dns_s3website_name": "", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dynamic_resharding": "true", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_gc_threads": "true", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_lc_threads": "true", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_ops_log": "false", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_quota_threads": "true", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_static_website": "false", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_usage_log": "false", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enforce_swift_acls": "true", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_exit_timeout_secs": "120", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_expose_bucket": "false", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_extended_http_attrs": "", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_filter": "none", 2024-05-06T00:26:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_frontends": "beast port=7480", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_concurrent_io": "10", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_deferred": "50", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_objs": "32", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_queue_size": "134213632", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_trim_chunk": "16", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_obj_min_wait": "7200", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_processor_max_time": "3600", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_processor_period": "3600", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_get_obj_window_size": "16777216", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_healthcheck_disabling_path": "", 2024-05-06T00:26:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ignore_get_invalid_range": "false", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_init_timeout": "300", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_kafka_connection_idle": "300", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_kafka_message_timeout": "5000", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_kafka_sleep_timeout": "10", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_domain": "", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_password": "", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_password_path": "", 2024-05-06T00:26:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_project": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_tenant": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_token": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_token_path": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_user": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_api_version": "2", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_domain": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_password": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_project": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_tenant": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_user": "", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_implicit_tenants": "false", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-05-06T00:26:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_service_token_enabled": "false", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_token_cache_size": "10000", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_url": "", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_verify_ssl": "true", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_debug_interval": "-1", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_lock_max_time": "90", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_max_objs": "32", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_max_rules": "1000", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_max_worker": "3", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_max_wp_worker": "3", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_thread_delay": "0", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_dnattr": "uid", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_searchfilter": "", 2024-05-06T00:26:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_uri": "", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lfuda_sync_frequency": "60", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_log_http_headers": "", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_log_nonexistent_bucket": "false", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_log_object_name_utc": "false", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-05-06T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_attr_name_len": "0", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_attr_size": "0", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_attrs_num_in_req": "0", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_chunk_size": "4194304", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_concurrent_requests": "1024", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_dynamic_shards": "1999", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_listing_results": "1000", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_notify_retries": "10", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_objs_per_shard": "100000", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_put_param_size": "1048576", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_put_size": "5368709120", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_slo_entries": "1000", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_md_log_max_shards": "64", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_md_notify_interval_msec": "200", 2024-05-06T00:26:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_meta_sync_poll_interval": "20", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_meta_sync_spawn_window": "20", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_mp_lock_max_time": "600", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_multipart_min_part_size": "5242880", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_fhcache_size": "2017", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_lru_lanes": "5", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_max_gc": "300", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-05-06T00:26:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_run_gc_threads": "false", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_run_lc_threads": "false", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_run_quota_threads": "false", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_run_sync_thread": "false", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_num_async_rados_threads": "32", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_num_control_oids": "8", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_numa_node": "-1", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_obj_stripe_size": "4194304", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_objexp_chunk_size": "100", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_objexp_gc_interval": "600", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_objexp_hints_num_shards": "127", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-05-06T00:26:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_op_thread_timeout": "600", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_op_tracing": "false", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_opa_token": "", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_opa_url": "", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_opa_verify_ssl": "true", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ops_log_rados": "false", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ops_log_socket_path": "", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_period_push_interval": "2.000000", 2024-05-06T00:26:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_period_push_interval_max": "30.000000", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_period_root_pool": ".rgw.root", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_cache_lanes": "3", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_cache_max_buckets": "100", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_cache_partitions": "3", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_print_continue": "true", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_print_prohibited_content_length": "false", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-05-06T00:26:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_rados_tracing": "false", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_realm": "", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_realm_id": "", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_region": "", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_region_root_pool": ".rgw.root", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_relaxed_region_enforcement": "false", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_request_uri": "", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_batch_size": "64", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_max_aio": "128", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_num_logs": "16", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_thread_interval": "600", 2024-05-06T00:26:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_resolve_cname": "false", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_rest_getusage_op_compat": "false", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_run_sync_thread": "true", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_use_keystone": "false", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_use_ldap": "false", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_use_rados": "true", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_use_sts": "false", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_success_create_obj_status": "0", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_scheduler_type": "throttler", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_script_uri": "", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_service_provider_name": "", 2024-05-06T00:26:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_client_id": "", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_client_secret": "", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_entry": "sts", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_key": "sts", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_max_session_duration": "43200", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_min_session_duration": "900", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_token_introspection_url": "", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_account_in_url": "false", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_auth_entry": "auth", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_auth_url": "", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_custom_header": "", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_enforce_content_length": "false", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_need_stats": "true", 2024-05-06T00:26:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_tenant_name": "", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_token_expiration": "86400", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_url": "", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_url_prefix": "swift", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_versioning_enabled": "false", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_lease_period": "120", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_log_trim_interval": "1200", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_obj_etag_verify": "false", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_trace_history_size": "4096", 2024-05-06T00:26:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_thread_pool_size": "512", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_topic_persistency_max_retries": "0", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_topic_require_publish_policy": "false", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_comment": "", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_createby": "", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_encoding": "", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_flag": "false", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_max_size": "5368709120", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_origin": "", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_sha_unit": "524288", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_tracker": "", 2024-05-06T00:26:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_trust_forwarded_https": "false", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_usage_log_tick_interval": "30", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_usage_max_shards": "32", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_usage_max_user_shards": "1", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_use_opa_authz": "false", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_counters_cache": "false", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_counters_cache_size": "10000", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_default_quota_max_size": "-1", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_max_buckets": "1000", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_policies_max_num": "100", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-05-06T00:26:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_quota_sync_interval": "86400", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_unique_email": "true", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_verify_ssl": "true", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_website_routing_rules_max_num": "50", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zone": "", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zone_id": "", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zonegroup": "", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zonegroup_id": "", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgwlc_auto_session_clear": "true", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rgwlc_skip_bucket_step": "false", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_block_size": "4096", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-05-06T00:26:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_shard_bits": "4", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_size": "536870912", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_type": "binned_lru", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_collect_compaction_stats": "false", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_collect_extended_stats": "false", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_collect_memory_stats": "false", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_index_type": "binary_search", 2024-05-06T00:26:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_log_to_ceph_log": "true", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_metadata_block_size": "4096", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_partition_filters": "false", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_perf": "false", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rotating_keys_renewal_timeout": "10", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "run_dir": "/var/run/ceph", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_block_create": "true", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_cache_lru_size": "67108864", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_cbjournal_size": "5368709120", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_data_delta_based_overwrite": "0", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_default_max_object_size": "16777216", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_device_size": "53687091200", 2024-05-06T00:26:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_full_integrity_check": "false", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_journal_batch_capacity": "16", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_journal_iodepth_limit": "5", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_main_device_type": "SSD", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_max_concurrent_transactions": "8", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_max_data_allocation_size": "32768", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_segment_size": "67108864", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "set_keepcaps": "false", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "setgroup": "", 2024-05-06T00:26:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "setuser": "", 2024-05-06T00:26:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "setuser_match_path": "", 2024-05-06T00:26:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_misplaced_ratio": "0.050000", 2024-05-06T00:26:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "thp": "false", 2024-05-06T00:26:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "threadpool_default_timeout": "60", 2024-05-06T00:26:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "threadpool_empty_queue_max_wait": "2", 2024-05-06T00:26:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "throttler_perf_counter": "true" 2024-05-06T00:26:42.314 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:42.316 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-05-06T00:26:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a: {} 2024-05-06T00:26:42.453 INFO:teuthology.orchestra.run.smithi078.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-06T00:26:42.466 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b: {} 2024-05-06T00:26:42.466 INFO:teuthology.orchestra.run.smithi078.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-06T00:26:42.475 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c: {} 2024-05-06T00:26:42.475 INFO:teuthology.orchestra.run.smithi078.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-06T00:26:42.491 DEBUG:teuthology.orchestra.run.smithi146:> type ceph-objectstore-tool 2024-05-06T00:26:42.496 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-05-06T00:26:42.496 DEBUG:teuthology.orchestra.run.smithi078:> type ceph-objectstore-tool 2024-05-06T00:26:42.500 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-05-06T00:26:42.501 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-05-06T00:26:42.512 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-05-06T00:26:42.512 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-05-06T00:26:42.512 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:26:42.512 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:26:42.513 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-06T00:26:42.513 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-05-06T00:26:42.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-05-06T00:26:42.513 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-05-06T00:26:42.513 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-05-06T00:26:42.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:26:42.514 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-05-06T00:26:42.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:26:42.514 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-05-06T00:26:42.515 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:26:42.515 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 512 -w 512 -t 512 -h 512 -l 250000000 -S 0 -N 6000 -M pool_client.0 image_client.0 2024-05-06T00:26:42.540 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:42.541+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:42.598 INFO:teuthology.orchestra.run.smithi186.stdout:mapped writes DISABLED 2024-05-06T00:26:42.598 INFO:teuthology.orchestra.run.smithi186.stdout:mapped reads DISABLED 2024-05-06T00:26:42.598 INFO:teuthology.orchestra.run.smithi186.stdout:Seed set to 4081 2024-05-06T00:26:42.598 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd mode enabled 2024-05-06T00:26:42.616 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:42.617+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:42.663 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:42.663 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:42.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:26:42.690 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:26:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:26:42.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:26:42.708 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:26:42.717 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:42.713+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:42.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:26:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:26:42.746 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:26:42.746 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:26:42.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:26:42.767 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:26:42.787 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:26:42.787 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:26:42.795 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:42.795 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:42.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:26:42.818 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:42.817+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:42.918 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:42.917+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:42.973 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:26:43.019 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:43.017+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:43.102 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:26:43.146 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:43.145+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:43.222 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:43.217+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:43.270 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:26:43.322 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:43.321+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:43.424 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:43.421+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:43.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:26:43.524 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:43.521+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:43.602 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:26:43.647 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:43.645+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:43.674 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-05-06T00:26:43.678 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:26:43.726 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:43.725+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:43.788 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:26:43.827 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:43.829+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:43.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:26:43.952 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:43.953+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:44.029 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:44.029+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:44.038 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:44.038 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:44.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:26:44.130 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:44.129+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:44.216 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:44.216 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:44.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:26:44.232 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:44.233+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:44.334 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:44.333+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:44.348 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:44.348 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:44.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:26:44.434 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:44.433+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:44.529 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:44.529 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:44.535 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:44.533+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:44.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:26:44.635 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:44.633+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:44.716 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:44.716 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:44.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:26:44.736 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:44.733+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:44.804 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:26:44.848 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:44.848 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:44.850 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:44.849+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:44.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:26:44.937 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:44.933+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:44.956 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:26:44.956 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:26:44.972 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:26:44.972 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:26:44.986 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:44.986 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:44.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:26:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:26:45.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:26:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:26:45.006 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:26:45.025 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:26:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:26:45.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:26:45.043 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:26:45.044 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:45.045+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:45.117 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:45.117 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:45.117 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:26:45.118 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:26:45.118 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:45.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:26:45.139 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:45.137+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:45.240 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:45.237+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:45.300 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:26:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:26:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [] 2024-05-06T00:26:45.301 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:45.315 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:26:45.362 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:45.361+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:45.442 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:45.441+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:45.443 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:45.443 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:45.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:26:45.546 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:45.545+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:45.622 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:45.622 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:45.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:26:45.678 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:45.677+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:45.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:26:45.766 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:45.765+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:45.808 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:18 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'8, mlcod=13'5)", 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.266801+0000", 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.541034636999999, 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003741711, 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.266801+0000", 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.266801+0000", 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.266803+0000", 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9199999999999998e-06 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.266851+0000", 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8495999999999998e-05 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.266880+0000", 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8912999999999999e-05 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.266893+0000", 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3030000000000001e-05 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.267016+0000", 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012318600000000001 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.267149+0000", 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013282600000000001 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.270467+0000", 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033180699999999998 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.270542+0000", 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5270000000000003e-05 2024-05-06T00:26:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:20 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.276005+0000", 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.531829939, 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027629450000000002, 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.276005+0000", 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.276005+0000", 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.276007+0000", 2024-05-06T00:26:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.88e-06 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.276267+0000", 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026003499999999999 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.276296+0000", 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8742999999999999e-05 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.276310+0000", 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3499999999999999e-05 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.276436+0000", 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012618899999999999 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.276490+0000", 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3842999999999999e-05 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278638+0000", 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021486529999999999 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278768+0000", 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130102 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.277378+0000", 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.530457693000001, 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036534890000000002, 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.277378+0000", 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.277378+0000", 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.277380+0000", 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9269999999999999e-06 2024-05-06T00:26:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.277515+0000", 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.277544+0000", 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8333000000000001e-05 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.277562+0000", 2024-05-06T00:26:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8170000000000001e-05 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.277722+0000", 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016058199999999999 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.277777+0000", 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4042999999999997e-05 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.280947+0000", 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003170586 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281031+0000", 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4096000000000003e-05 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:22 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-05-06T00:26:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.278485+0000", 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.529350665000001, 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026253040000000002, 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278485+0000", 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278485+0000", 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278487+0000", 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.854e-06 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278608+0000", 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121332 2024-05-06T00:26:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278632+0000", 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3736000000000001e-05 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278644+0000", 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.276e-05 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278890+0000", 2024-05-06T00:26:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024556799999999999 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.278945+0000", 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5189999999999998e-05 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281063+0000", 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021179269999999999 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281110+0000", 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6937000000000001e-05 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:23 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.279833+0000", 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.528002849, 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044081570000000002, 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279833+0000", 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279833+0000", 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279834+0000", 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.874e-06 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279949+0000", 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011439599999999999 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279978+0000", 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9386e-05 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279992+0000", 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3613e-05 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.280135+0000", 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014314899999999999 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.280193+0000", 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8122999999999999e-05 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.284143+0000", 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039495930000000004 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.284241+0000", 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8023000000000007e-05 2024-05-06T00:26:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:24 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'14, mlcod=13'9)", 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.281389+0000", 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.526446557, 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003497787, 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281389+0000", 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281389+0000", 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281390+0000", 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281469+0000", 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8769000000000004e-05 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281496+0000", 2024-05-06T00:26:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.726e-05 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281510+0000", 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3337e-05 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281605+0000", 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5789000000000001e-05 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.281638+0000", 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2382999999999998e-05 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.284804+0000", 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031661150000000002 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.284887+0000", 2024-05-06T00:26:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2806999999999995e-05 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:49 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'31, mlcod=13'26)", 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.393678+0000", 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.414157224, 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039713789999999997, 2024-05-06T00:26:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.393678+0000", 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.393678+0000", 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.393679+0000", 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.043e-06 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.393686+0000", 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8140000000000003e-06 2024-05-06T00:26:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.393699+0000", 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.307e-05 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.393706+0000", 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5300000000000002e-06 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.393756+0000", 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0555e-05 2024-05-06T00:26:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.393775+0000", 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8539e-05 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.397608+0000", 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038335209999999999 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.397650+0000", 2024-05-06T00:26:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1307000000000003e-05 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:50 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'27)", 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.395131+0000", 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.412704680999999, 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054106950000000001, 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395131+0000", 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395131+0000", 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395132+0000", 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1519999999999999e-06 2024-05-06T00:26:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395203+0000", 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1229000000000005e-05 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395220+0000", 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6741000000000001e-05 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395249+0000", 2024-05-06T00:26:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9300000000000001e-05 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395310+0000", 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0782000000000002e-05 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395341+0000", 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1424000000000003e-05 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.400493+0000", 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051515490000000001 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.400541+0000", 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8517999999999999e-05 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:51 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'27)", 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.396451+0000", 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.411384903, 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041225380000000002, 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.396451+0000", 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.396451+0000", 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.396452+0000", 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1379999999999999e-06 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.396472+0000", 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0721999999999999e-05 2024-05-06T00:26:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.396489+0000", 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6932e-05 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.396500+0000", 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.396557+0000", 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7000999999999999e-05 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.396584+0000", 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7253000000000001e-05 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.400543+0000", 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039596220000000003 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.400573+0000", 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9672e-05 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:52 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'34, mlcod=13'29)", 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.397966+0000", 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.409869787, 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030475419999999999, 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.397966+0000", 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.397966+0000", 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.397966+0000", 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8100000000000002e-07 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.397973+0000", 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.397989+0000", 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6036999999999999e-05 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.398000+0000", 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0665999999999999e-05 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.398053+0000", 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3504999999999997e-05 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.398069+0000", 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5892999999999998e-05 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.400980+0000", 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029108160000000001 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.401013+0000", 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3370000000000001e-05 2024-05-06T00:26:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:72 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'50, mlcod=13'46)", 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.590000+0000", 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.217835048, 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031885450000000001, 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590000+0000", 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590000+0000", 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590002+0000", 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9429999999999999e-06 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590128+0000", 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012524899999999999 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590155+0000", 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7042999999999998e-05 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590167+0000", 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2863e-05 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590246+0000", 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8076000000000003e-05 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590274+0000", 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8887e-05 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593088+0000", 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028138479999999999 2024-05-06T00:26:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593189+0000", 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100636 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:73 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'51, mlcod=13'46)", 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.590761+0000", 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.217074261, 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002513993, 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590761+0000", 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590761+0000", 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590763+0000", 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8530000000000001e-06 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590775+0000", 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1667e-05 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590798+0000", 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2963e-05 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.590833+0000", 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5116000000000001e-05 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.591270+0000", 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043713699999999999 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.591323+0000", 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2669e-05 2024-05-06T00:26:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593196+0000", 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018731189999999999 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593275+0000", 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9468999999999994e-05 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:74 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'52, mlcod=13'48)", 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.591789+0000", 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.216046902, 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039589350000000002, 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.591789+0000", 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.591789+0000", 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.591790+0000", 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.846e-06 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.591804+0000", 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.394e-05 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.591827+0000", 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3139999999999999e-05 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.591840+0000", 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.287e-05 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.592212+0000", 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037150300000000002 2024-05-06T00:26:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.592294+0000", 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1983000000000004e-05 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.595643+0000", 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033488239999999998 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.595747+0000", 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104829 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:75 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'48)", 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.592415+0000", 2024-05-06T00:26:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.215420720999999, 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037197110000000001, 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.592415+0000", 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.592415+0000", 2024-05-06T00:26:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.592417+0000", 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.844e-06 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.592474+0000", 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7509000000000001e-05 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.592498+0000", 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3580000000000001e-05 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.592510+0000", 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.275e-05 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593401+0000", 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089076599999999998 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593490+0000", 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9073000000000006e-05 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.596049+0000", 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025585970000000001 2024-05-06T00:26:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.596134+0000", 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5592000000000003e-05 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:76 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'48)", 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.593331+0000", 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.214504850999999, 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034874659999999998, 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593331+0000", 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593331+0000", 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593332+0000", 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593343+0000", 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0873000000000001e-05 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593365+0000", 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2580000000000001e-05 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.593383+0000", 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7569999999999999e-05 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.594966+0000", 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001583557 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.595074+0000", 2024-05-06T00:26:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010710899999999999 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.596759+0000", 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016853110000000001 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.596818+0000", 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9166000000000002e-05 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:77 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'55, mlcod=13'53)", 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.597870+0000", 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.209965099, 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003159589, 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.597870+0000", 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.597870+0000", 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.597872+0000", 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5999999999999999e-06 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.597985+0000", 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011325200000000001 2024-05-06T00:26:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598005+0000", 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9833000000000001e-05 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598018+0000", 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2809999999999999e-05 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598130+0000", 2024-05-06T00:26:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000111703 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598160+0000", 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0956e-05 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.600944+0000", 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027833020000000001 2024-05-06T00:26:45.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.601030+0000", 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6132999999999997e-05 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:78 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'56, mlcod=13'53)", 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.598481+0000", 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.209354111, 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034860300000000002, 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598481+0000", 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598481+0000", 2024-05-06T00:26:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598483+0000", 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7039999999999999e-06 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598645+0000", 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016163199999999999 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598672+0000", 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7120000000000001e-05 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598688+0000", 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6636000000000001e-05 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598943+0000", 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025419800000000002 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.598997+0000", 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4716999999999998e-05 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.601864+0000", 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00286689 2024-05-06T00:26:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.601967+0000", 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103133 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:79 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'57, mlcod=13'53)", 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.599484+0000", 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.208351171, 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025745659999999999, 2024-05-06T00:26:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.599484+0000", 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.599484+0000", 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.599486+0000", 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9659999999999999e-06 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.599616+0000", 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013018599999999999 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.599639+0000", 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2802999999999998e-05 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.599656+0000", 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6776e-05 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.600048+0000", 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039184099999999998 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.600099+0000", 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1669000000000003e-05 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.601979+0000", 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001879309 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.602059+0000", 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0016000000000001e-05 2024-05-06T00:26:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:45.731342+0000", 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.076493285999999994, 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035692689999999999, 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:26:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731342+0000", 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731342+0000", 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731345+0000", 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6230000000000001e-06 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731358+0000", 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3349999999999999e-05 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731379+0000", 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1100000000000001e-05 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731395+0000", 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5892999999999998e-05 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731483+0000", 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7472999999999994e-05 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731650+0000", 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016762799999999999 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.731894+0000", 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024352999999999999 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.734446+0000", 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00255244 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.734757+0000", 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031123700000000002 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.734838+0000", 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0814000000000002e-05 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.734911+0000", 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3181000000000003e-05 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:15 3.1f 3:fa654f18:::rbd_id.image_client.0:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:45.748382+0000", 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.059453842, 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032989880000000001, 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 15 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.748382+0000", 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.748382+0000", 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.748383+0000", 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2049999999999999e-06 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.748389+0000", 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3949999999999998e-06 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.748404+0000", 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5108e-05 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.748411+0000", 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.748463+0000", 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1564999999999998e-05 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.748556+0000", 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2464000000000001e-05 2024-05-06T00:26:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.751681+0000", 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031250539999999999 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:45.905 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:45.907 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:26:45.907 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:45.905+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:46.008 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:46.009+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:46.073 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:46.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:46.073 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:46.073 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:46.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:26:46.109 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:46.109+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:46.207 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:46.207 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:46.210 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:46.209+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:46.220 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:26:46.310 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:46.309+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:46.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:26:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:26:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:26:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4364.0:9 3.1a e21/18 3:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-05-06T00:26:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:45.719066+0000", 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.62640495900000004, 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00241522, 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.719066+0000", 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.719066+0000", 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.719066+0000", 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7400000000000003e-07 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.719076+0000", 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0206e-05 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.719098+0000", 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.126e-05 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.719114+0000", 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6779999999999999e-05 2024-05-06T00:26:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.719209+0000", 2024-05-06T00:26:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4738e-05 2024-05-06T00:26:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.719241+0000", 2024-05-06T00:26:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2178999999999998e-05 2024-05-06T00:26:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.721423+0000", 2024-05-06T00:26:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021814590000000002 2024-05-06T00:26:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:45.721481+0000", 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8023999999999999e-05 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:46.352 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:46.360 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:26:46.411 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:46.409+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:46.512 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:46.513+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:46.525 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:46.525 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:46.539 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:26:46.613 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:46.613+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:46.657 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:46.657 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:46.658 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:46.658 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:46.658 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:46.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:26:46.710 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:26:46.714 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:46.713+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:26:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:26:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [] 2024-05-06T00:26:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:46.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:26:46.856 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:46.857+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:46.916 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:46.917+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:46.941 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:46.942 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:46.942 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:26:46.942 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:46.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:26:47.044 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:47.045+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:47.060 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:26:47.118 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:26:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:26:47.119 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:47.119 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:47.117+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:47.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:26:47.213 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:26:47.213 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:26:47.219 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:47.217+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:47.229 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:26:47.230 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:26:47.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:26:47.246 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:26:47.264 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:26:47.265 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:26:47.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:26:47.285 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:26:47.298 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:47.299 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:47.305 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:26:47.305 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:26:47.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:26:47.360 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:47.361+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:47.420 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:47.417+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:47.433 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:47.433 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:47.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:26:47.522 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:47.521+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:47.610 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:47.610 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:47.623 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:47.621+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:47.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:26:47.724 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:47.721+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:47.796 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:47.796 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:47.812 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:26:47.820 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0x4be0e00 2024-05-06T00:26:47.827 INFO:teuthology.orchestra.run.smithi186.stdout:2 trunc from 0x0 to 0x4be0e00 2024-05-06T00:26:47.860 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:47.861+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:47.871 INFO:teuthology.orchestra.run.smithi186.stdout:8 write 0xb08b600 thru 0xb08fc3c (0x463d bytes) 2024-05-06T00:26:47.891 INFO:teuthology.orchestra.run.smithi186.stdout:9 trunc from 0xb08fe00 to 0x2b38200 2024-05-06T00:26:47.923 INFO:teuthology.orchestra.run.smithi186.stdout:10 punch from 0x16a5600 to 0x16b2400, (0xce00 bytes) 2024-05-06T00:26:47.925 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:47.925+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:47.950 INFO:teuthology.orchestra.run.smithi186.stdout:11 write 0x755c400 thru 0x7566f2c (0xab2d bytes) 2024-05-06T00:26:47.950 INFO:teuthology.orchestra.run.smithi186.stdout:13 read 0x4611400 thru 0x461d96a (0xc56b bytes) 2024-05-06T00:26:47.950 INFO:teuthology.orchestra.run.smithi186.stdout:16 clone 1 order 24 su 4194304 sc 8 2024-05-06T00:26:47.990 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:47.990 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:48.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:26:48.026 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:48.025+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:48.126 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:48.125+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:48.173 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:48.173 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:48.187 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:26:48.227 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:48.225+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:48.303 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:48.304 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:48.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:26:48.327 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:48.329+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:48.429 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:48.429+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:48.449 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:48.449 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:48.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:26:48.529 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:48.529+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:48.585 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:48.585 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:48.599 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:26:48.630 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:48.629+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:48.715 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:26:48.716 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:26:48.716 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-06T00:26:48.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-06T00:26:48.717 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:48.718 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:48.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:26:48.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:26:48.735 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:48.733+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:48.831 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:48.829+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:48.900 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:48.900 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:48.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:26:48.962 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:48.961+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 23, 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b583c614-b2ca-4769-90c0-44726bf69b34", 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-06T00:26:16.923985+0000", 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-06T00:26:48.713276+0000", 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-06T00:26:27.394947+0000", 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-06T00:26:18.750504+0000", 2024-05-06T00:26:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5806080, 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "nodeep-scrub", 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "noscrub", 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-06T00:26:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-06T00:26:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-06T00:26:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-06T00:26:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-06T00:26:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-06T00:26:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 6, 2024-05-06T00:26:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-06T00:26:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "jewel", 2024-05-06T00:26:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:28.436494+0000", 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:26:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:26:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "14", 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:26:49.020 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:26:49.021 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:26:49.022 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 6.059999942779541, 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 6.059999942779541, 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.33000001311302185, 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 1, 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 1 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-06T00:26:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-06T00:26:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:33.941780+0000", 2024-05-06T00:26:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:26:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:26:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:26:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:26:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-06T00:26:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-06T00:26:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-06T00:26:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-06T00:26:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:26:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:26:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:26:49.029 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:26:49.029 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:26:49.029 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:26:49.029 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:26:49.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.030 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-06T00:26:49.030 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:26:49.030 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:26:49.030 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:26:49.030 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:26:49.031 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:26:49.032 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:26:49.032 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:26:49.032 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:26:49.032 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:26:49.032 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:26:49.032 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:26:49.033 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.25, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.25, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 1, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2.25, 2024-05-06T00:26:49.034 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2.25, 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 1, 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 1 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:42.622595+0000", 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:26:49.035 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:26:49.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:26:49.036 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:26:49.036 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:26:49.036 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:26:49.036 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:26:49.036 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:26:49.036 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:26:49.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "23", 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 3, 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 23, 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:26:49.038 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:26:49.039 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:26:49.039 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:26:49.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:26:49.040 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:26:49.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:26:49.042 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 1.5, 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 1.5, 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 1, 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.5, 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.5, 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 1, 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 1 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.043 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d31b9a6a-864b-4c24-99db-8cc6cf3a5c24", 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:49.044 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-06T00:26:49.045 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.047 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-06T00:26:49.047 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:49.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.047 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.047 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:49.047 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:49.048 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6817/1427564096", 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6819/1427564096", 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6823/1427564096", 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6821/1427564096", 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "86509826-0e53-4678-8260-17c97c05af54", 2024-05-06T00:26:49.050 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:49.053 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.055 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-06T00:26:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6803/2663808859", 2024-05-06T00:26:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6807/2663808859", 2024-05-06T00:26:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/2663808859", 2024-05-06T00:26:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6811/2663808859", 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "8973b524-521e-41c0-87f2-85cb390fbfeb", 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:49.057 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:49.059 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.062 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6802/385938386", 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6806/385938386", 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6814/385938386", 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6810/385938386", 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "3b0d07bd-132e-48ff-83ac-7182eee05df9", 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:49.064 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.065 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6811", 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6812", 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:49.066 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6814", 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6816", 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.068 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:49.068 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.068 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6821", 2024-05-06T00:26:49.068 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:49.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6822", 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6817", 2024-05-06T00:26:49.069 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6819", 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6812/4178038317", 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6816/4178038317", 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6822/4178038317", 2024-05-06T00:26:49.070 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6819/4178038317", 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2", 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:49.071 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6808", 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.072 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6809", 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6810", 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.073 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6813", 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6820", 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6823", 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6815", 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:49.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6818", 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6809/1052555350", 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6813/1052555350", 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1052555350", 2024-05-06T00:26:49.076 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6818/1052555350", 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "34b61f5f-e407-4414-8bc5-11263bbe261a", 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.079 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6800", 2024-05-06T00:26:49.079 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:49.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.079 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6801", 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6802", 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.080 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6803", 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6806", 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6807", 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:49.082 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6804", 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6805", 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6801/2019146845", 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6803/2019146845", 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6807/2019146845", 2024-05-06T00:26:49.083 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6805/2019146845", 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.276988+0000", 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.054473+0000", 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.086 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:26:49.086 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:49.086 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:49.086 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:49.086 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:49.086 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:49.086 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.154558+0000", 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.284746+0000", 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.259847+0000", 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:49.088 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.180375+0000", 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [], 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [], 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [], 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-06T00:26:49.089 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-06T00:26:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-06T00:26:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-06T00:26:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-06T00:26:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-06T00:26:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-06T00:26:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-06T00:26:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-06T00:26:49.090 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [ 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 2, 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 1 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:49.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [], 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:49.092 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:49.093 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:49.094 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:49.095 INFO:tasks.thrashosds.thrasher:j is {'epoch': 23, 'fsid': 'b583c614-b2ca-4769-90c0-44726bf69b34', 'created': '2024-05-06T00:26:16.923985+0000', 'modified': '2024-05-06T00:26:48.713276+0000', 'last_up_change': '2024-05-06T00:26:27.394947+0000', 'last_in_change': '2024-05-06T00:26:18.750504+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T00:26:28.436494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-05-06T00:26:33.941780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-06T00:26:42.622595+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 23, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd31b9a6a-864b-4c24-99db-8cc6cf3a5c24', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6817', 'nonce': 1427564096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6818', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 1427564096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6822', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 1427564096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6821', 'nonce': 1427564096}]}, 'public_addr': '172.21.15.78:6817/1427564096', 'cluster_addr': '172.21.15.78:6819/1427564096', 'heartbeat_back_addr': '172.21.15.78:6823/1427564096', 'heartbeat_front_addr': '172.21.15.78:6821/1427564096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '86509826-0e53-4678-8260-17c97c05af54', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6801', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6803', 'nonce': 2663808859}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6805', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6807', 'nonce': 2663808859}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6813', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 2663808859}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6809', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6811', 'nonce': 2663808859}]}, 'public_addr': '172.21.15.78:6803/2663808859', 'cluster_addr': '172.21.15.78:6807/2663808859', 'heartbeat_back_addr': '172.21.15.78:6815/2663808859', 'heartbeat_front_addr': '172.21.15.78:6811/2663808859', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8973b524-521e-41c0-87f2-85cb390fbfeb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6802', 'nonce': 385938386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6804', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6806', 'nonce': 385938386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6812', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6814', 'nonce': 385938386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6808', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6810', 'nonce': 385938386}]}, 'public_addr': '172.21.15.78:6802/385938386', 'cluster_addr': '172.21.15.78:6806/385938386', 'heartbeat_back_addr': '172.21.15.78:6814/385938386', 'heartbeat_front_addr': '172.21.15.78:6810/385938386', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3b0d07bd-132e-48ff-83ac-7182eee05df9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6811', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6812', 'nonce': 4178038317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6816', 'nonce': 4178038317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6821', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6822', 'nonce': 4178038317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6817', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 4178038317}]}, 'public_addr': '172.21.15.146:6812/4178038317', 'cluster_addr': '172.21.15.146:6816/4178038317', 'heartbeat_back_addr': '172.21.15.146:6822/4178038317', 'heartbeat_front_addr': '172.21.15.146:6819/4178038317', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 1052555350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 1052555350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1052555350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6815', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6818', 'nonce': 1052555350}]}, 'public_addr': '172.21.15.146:6809/1052555350', 'cluster_addr': '172.21.15.146:6813/1052555350', 'heartbeat_back_addr': '172.21.15.146:6823/1052555350', 'heartbeat_front_addr': '172.21.15.146:6818/1052555350', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '34b61f5f-e407-4414-8bc5-11263bbe261a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 2019146845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 2019146845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 2019146845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 2019146845}]}, 'public_addr': '172.21.15.146:6801/2019146845', 'cluster_addr': '172.21.15.146:6803/2019146845', 'heartbeat_back_addr': '172.21.15.146:6807/2019146845', 'heartbeat_front_addr': '172.21.15.146:6805/2019146845', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.276988+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.054473+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.154558+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.284746+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.259847+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.180375+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-06T00:26:49.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:26:49.096 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:26:49.097 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:49.093+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:49.195 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:49.193+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:49.203 INFO:teuthology.orchestra.run.smithi186.stdout:20 read 0x589e200 thru 0x58a6fef (0x8df0 bytes) 2024-05-06T00:26:49.205 INFO:teuthology.orchestra.run.smithi186.stdout:21 punch from 0x4548000 to 0x4551a00, (0x9a00 bytes) 2024-05-06T00:26:49.251 INFO:teuthology.orchestra.run.smithi186.stdout:28 trunc from 0x7567000 to 0x2651200 2024-05-06T00:26:49.268 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:49.268 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:49.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:26:49.291 INFO:teuthology.orchestra.run.smithi186.stdout:29 punch from 0x1daee00 to 0x1db7400, (0x8600 bytes) 2024-05-06T00:26:49.294 INFO:teuthology.orchestra.run.smithi186.stdout:32 write 0xd35e00 thru 0xd3900a (0x320b bytes) 2024-05-06T00:26:49.295 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:49.293+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:49.295 INFO:teuthology.orchestra.run.smithi186.stdout:33 read 0x1936a00 thru 0x193fa8c (0x908d bytes) 2024-05-06T00:26:49.296 INFO:teuthology.orchestra.run.smithi186.stdout:35 read 0x5ab800 thru 0x5b5c88 (0xa489 bytes) 2024-05-06T00:26:49.323 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:26:49.346 INFO:teuthology.orchestra.run.smithi186.stdout:36 write 0xb0e9a00 thru 0xb0f23da (0x89db bytes) 2024-05-06T00:26:49.348 INFO:teuthology.orchestra.run.smithi186.stdout:37 punch from 0xa7ce600 to 0xa7d3400, (0x4e00 bytes) 2024-05-06T00:26:49.348 INFO:teuthology.orchestra.run.smithi186.stdout:38 write 0x2542400 thru 0x254e1ad (0xbdae bytes) 2024-05-06T00:26:49.349 INFO:teuthology.orchestra.run.smithi186.stdout:39 read 0x52dd200 thru 0x52df464 (0x2265 bytes) 2024-05-06T00:26:49.350 INFO:teuthology.orchestra.run.smithi186.stdout:40 read 0xa935e00 thru 0xa93eaa7 (0x8ca8 bytes) 2024-05-06T00:26:49.350 INFO:teuthology.orchestra.run.smithi186.stdout:41 read 0x2356800 thru 0x2357a55 (0x1256 bytes) 2024-05-06T00:26:49.351 INFO:teuthology.orchestra.run.smithi186.stdout:42 read 0xa05fa00 thru 0xa05fc9d (0x29e bytes) 2024-05-06T00:26:49.351 INFO:teuthology.orchestra.run.smithi186.stdout:43 write 0x8eff600 thru 0x8f05cea (0x66eb bytes) 2024-05-06T00:26:49.389 INFO:teuthology.orchestra.run.smithi186.stdout:45 write 0xdcd0e00 thru 0xdcd5102 (0x4303 bytes) 2024-05-06T00:26:49.391 INFO:teuthology.orchestra.run.smithi186.stdout:49 read 0xbecca00 thru 0xbedbfe4 (0xf5e5 bytes) 2024-05-06T00:26:49.391 INFO:teuthology.orchestra.run.smithi186.stdout:52 read 0xfe1800 thru 0xfeebe1 (0xd3e2 bytes) 2024-05-06T00:26:49.392 INFO:teuthology.orchestra.run.smithi186.stdout:54 write 0x86d8000 thru 0x86e6676 (0xe677 bytes) 2024-05-06T00:26:49.393 INFO:teuthology.orchestra.run.smithi186.stdout:57 read 0xcde5c00 thru 0xcdebd0a (0x610b bytes) 2024-05-06T00:26:49.394 INFO:teuthology.orchestra.run.smithi186.stdout:63 write 0x9d7f600 thru 0x9d8ecb3 (0xf6b4 bytes) 2024-05-06T00:26:49.394 INFO:teuthology.orchestra.run.smithi186.stdout:64 read 0x97cee00 thru 0x97d52a8 (0x64a9 bytes) 2024-05-06T00:26:49.394 INFO:teuthology.orchestra.run.smithi186.stdout:66 read 0x2d64000 thru 0x2d64fa3 (0xfa4 bytes) 2024-05-06T00:26:49.394 INFO:teuthology.orchestra.run.smithi186.stdout:67 write 0x2e68200 thru 0x2e6fb46 (0x7947 bytes) 2024-05-06T00:26:49.395 INFO:teuthology.orchestra.run.smithi186.stdout:69 write 0xd63ca00 thru 0xd64a17a (0xd77b bytes) 2024-05-06T00:26:49.395 INFO:teuthology.orchestra.run.smithi186.stdout:70 punch from 0x8160400 to 0x8166e00, (0x6a00 bytes) 2024-05-06T00:26:49.396 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:49.393+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:49.410 INFO:teuthology.orchestra.run.smithi186.stdout:71 trunc from 0xdcd5200 to 0x3650800 2024-05-06T00:26:49.425 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:26:49.454 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0x70d3000 2024-05-06T00:26:49.457 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:26:49.460 INFO:teuthology.orchestra.run.smithi186.stdout:73 trunc from 0x3650800 to 0x70d3000 2024-05-06T00:26:49.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:26:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:26:49.476 INFO:teuthology.orchestra.run.smithi186.stdout:74 read 0xcb4200 thru 0xcb9543 (0x5344 bytes) 2024-05-06T00:26:49.477 INFO:teuthology.orchestra.run.smithi186.stdout:77 punch from 0x6569200 to 0x6573200, (0xa000 bytes) 2024-05-06T00:26:49.478 INFO:teuthology.orchestra.run.smithi186.stdout:79 read 0x2118600 thru 0x2120e7c (0x887d bytes) 2024-05-06T00:26:49.479 INFO:teuthology.orchestra.run.smithi186.stdout:80 punch from 0x4b45600 to 0x4b4ac00, (0x5600 bytes) 2024-05-06T00:26:49.479 INFO:teuthology.orchestra.run.smithi186.stdout:81 read 0x244d400 thru 0x245269f (0x52a0 bytes) 2024-05-06T00:26:49.479 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-05-06T00:26:48.725290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590433,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":130,"num_read_kb":106,"num_write":107,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164588,"kb_used_data":2444,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066452,"statfs":{"total":579820584960,"available":579652046848,"internally_reserved":0,"allocated":2502656,"data_stored":1634013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":64,"num_read_kb":51,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.328270"},"pg_stats":[{"pgid":"3.1f","version":"21'1","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.843108+0000","last_change":"2024-05-06T00:26:45.712771+0000","last_active":"2024-05-06T00:26:45.843108+0000","last_peered":"2024-05-06T00:26:45.843108+0000","last_clean":"2024-05-06T00:26:45.843108+0000","last_became_active":"2024-05-06T00:26:43.694494+0000","last_became_peered":"2024-05-06T00:26:43.694494+0000","last_unstale":"2024-05-06T00:26:45.843108+0000","last_undegraded":"2024-05-06T00:26:45.843108+0000","last_fullsized":"2024-05-06T00:26:45.843108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:57:19.572248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.715971+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:26:46.715971+0000","last_peered":"2024-05-06T00:26:46.715971+0000","last_clean":"2024-05-06T00:26:46.715971+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:26:46.715971+0000","last_undegraded":"2024-05-06T00:26:46.715971+0000","last_fullsized":"2024-05-06T00:26:46.715971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:11.329622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.733037+0000","last_change":"2024-05-06T00:26:45.733037+0000","last_active":"2024-05-06T00:26:45.733037+0000","last_peered":"2024-05-06T00:26:45.733037+0000","last_clean":"2024-05-06T00:26:45.733037+0000","last_became_active":"2024-05-06T00:26:43.711475+0000","last_became_peered":"2024-05-06T00:26:43.711475+0000","last_unstale":"2024-05-06T00:26:45.733037+0000","last_undegraded":"2024-05-06T00:26:45.733037+0000","last_fullsized":"2024-05-06T00:26:45.733037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:24:34.347783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"21'1","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.732556+0000","last_change":"2024-05-06T00:26:45.732216+0000","last_active":"2024-05-06T00:26:45.732556+0000","last_peered":"2024-05-06T00:26:45.732556+0000","last_clean":"2024-05-06T00:26:45.732556+0000","last_became_active":"2024-05-06T00:26:43.715832+0000","last_became_peered":"2024-05-06T00:26:43.715832+0000","last_unstale":"2024-05-06T00:26:45.732556+0000","last_undegraded":"2024-05-06T00:26:45.732556+0000","last_fullsized":"2024-05-06T00:26:45.732556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:55:13.093657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.713563+0000","last_change":"2024-05-06T00:26:45.713563+0000","last_active":"2024-05-06T00:26:45.713563+0000","last_peered":"2024-05-06T00:26:45.713563+0000","last_clean":"2024-05-06T00:26:45.713563+0000","last_became_active":"2024-05-06T00:26:43.707253+0000","last_became_peered":"2024-05-06T00:26:43.707253+0000","last_unstale":"2024-05-06T00:26:45.713563+0000","last_undegraded":"2024-05-06T00:26:45.713563+0000","last_fullsized":"2024-05-06T00:26:45.713563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:18:38.559702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"21'2","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.883364+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:26:45.883364+0000","last_peered":"2024-05-06T00:26:45.883364+0000","last_clean":"2024-05-06T00:26:45.883364+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:26:45.883364+0000","last_undegraded":"2024-05-06T00:26:45.883364+0000","last_fullsized":"2024-05-06T00:26:45.883364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:49:05.872292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716270+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:26:46.716270+0000","last_peered":"2024-05-06T00:26:46.716270+0000","last_clean":"2024-05-06T00:26:46.716270+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:26:46.716270+0000","last_undegraded":"2024-05-06T00:26:46.716270+0000","last_fullsized":"2024-05-06T00:26:46.716270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:36:49.623551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.715742+0000","last_change":"2024-05-06T00:26:45.713666+0000","last_active":"2024-05-06T00:26:46.715742+0000","last_peered":"2024-05-06T00:26:46.715742+0000","last_clean":"2024-05-06T00:26:46.715742+0000","last_became_active":"2024-05-06T00:26:43.688284+0000","last_became_peered":"2024-05-06T00:26:43.688284+0000","last_unstale":"2024-05-06T00:26:46.715742+0000","last_undegraded":"2024-05-06T00:26:46.715742+0000","last_fullsized":"2024-05-06T00:26:46.715742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:55:37.811526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"21'1","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.741204+0000","last_change":"2024-05-06T00:26:45.713245+0000","last_active":"2024-05-06T00:26:45.741204+0000","last_peered":"2024-05-06T00:26:45.741204+0000","last_clean":"2024-05-06T00:26:45.741204+0000","last_became_active":"2024-05-06T00:26:43.698443+0000","last_became_peered":"2024-05-06T00:26:43.698443+0000","last_unstale":"2024-05-06T00:26:45.741204+0000","last_undegraded":"2024-05-06T00:26:45.741204+0000","last_fullsized":"2024-05-06T00:26:45.741204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:16:27.766000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":30,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.733718+0000","last_change":"2024-05-06T00:26:45.733718+0000","last_active":"2024-05-06T00:26:45.733718+0000","last_peered":"2024-05-06T00:26:45.733718+0000","last_clean":"2024-05-06T00:26:45.733718+0000","last_became_active":"2024-05-06T00:26:43.703465+0000","last_became_peered":"2024-05-06T00:26:43.703465+0000","last_unstale":"2024-05-06T00:26:45.733718+0000","last_undegraded":"2024-05-06T00:26:45.733718+0000","last_fullsized":"2024-05-06T00:26:45.733718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:10:30.357213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.717341+0000","last_change":"2024-05-06T00:26:45.711104+0000","last_active":"2024-05-06T00:26:46.717341+0000","last_peered":"2024-05-06T00:26:46.717341+0000","last_clean":"2024-05-06T00:26:46.717341+0000","last_became_active":"2024-05-06T00:26:43.692137+0000","last_became_peered":"2024-05-06T00:26:43.692137+0000","last_unstale":"2024-05-06T00:26:46.717341+0000","last_undegraded":"2024-05-06T00:26:46.717341+0000","last_fullsized":"2024-05-06T00:26:46.717341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:25:50.559250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001868853,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.717710+0000","last_change":"2024-05-06T00:26:45.710285+0000","last_active":"2024-05-06T00:26:46.717710+0000","last_peered":"2024-05-06T00:26:46.717710+0000","last_clean":"2024-05-06T00:26:46.717710+0000","last_became_active":"2024-05-06T00:26:43.706306+0000","last_became_peered":"2024-05-06T00:26:43.706306+0000","last_unstale":"2024-05-06T00:26:46.717710+0000","last_undegraded":"2024-05-06T00:26:46.717710+0000","last_fullsized":"2024-05-06T00:26:46.717710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:05:16.441602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.717900+0000","last_change":"2024-05-06T00:26:36.150377+0000","last_active":"2024-05-06T00:26:46.717900+0000","last_peered":"2024-05-06T00:26:46.717900+0000","last_clean":"2024-05-06T00:26:46.717900+0000","last_became_active":"2024-05-06T00:26:36.149496+0000","last_became_peered":"2024-05-06T00:26:36.149496+0000","last_unstale":"2024-05-06T00:26:46.717900+0000","last_undegraded":"2024-05-06T00:26:46.717900+0000","last_fullsized":"2024-05-06T00:26:46.717900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:54.410549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.712584+0000","last_change":"2024-05-06T00:26:45.712584+0000","last_active":"2024-05-06T00:26:45.712584+0000","last_peered":"2024-05-06T00:26:45.712584+0000","last_clean":"2024-05-06T00:26:45.712584+0000","last_became_active":"2024-05-06T00:26:43.688322+0000","last_became_peered":"2024-05-06T00:26:43.688322+0000","last_unstale":"2024-05-06T00:26:45.712584+0000","last_undegraded":"2024-05-06T00:26:45.712584+0000","last_fullsized":"2024-05-06T00:26:45.712584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:59:23.742660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716758+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:26:46.716758+0000","last_peered":"2024-05-06T00:26:46.716758+0000","last_clean":"2024-05-06T00:26:46.716758+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:26:46.716758+0000","last_undegraded":"2024-05-06T00:26:46.716758+0000","last_fullsized":"2024-05-06T00:26:46.716758+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716092+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:26:46.716092+0000","last_peered":"2024-05-06T00:26:46.716092+0000","last_clean":"2024-05-06T00:26:46.716092+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:26:46.716092+0000","last_undegraded":"2024-05-06T00:26:46.716092+0000","last_fullsized":"2024-05-06T00:26:46.716092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:10:33.060913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.717689+0000","last_change":"2024-05-06T00:26:35.642448+0000","last_active":"2024-05-06T00:26:46.717689+0000","last_peered":"2024-05-06T00:26:46.717689+0000","last_clean":"2024-05-06T00:26:46.717689+0000","last_became_active":"2024-05-06T00:26:35.641678+0000","last_became_peered":"2024-05-06T00:26:35.641678+0000","last_unstale":"2024-05-06T00:26:46.717689+0000","last_undegraded":"2024-05-06T00:26:46.717689+0000","last_fullsized":"2024-05-06T00:26:46.717689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:32:32.988296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716557+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:26:46.716557+0000","last_peered":"2024-05-06T00:26:46.716557+0000","last_clean":"2024-05-06T00:26:46.716557+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:26:46.716557+0000","last_undegraded":"2024-05-06T00:26:46.716557+0000","last_fullsized":"2024-05-06T00:26:46.716557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:35:29.972870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716615+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:26:46.716615+0000","last_peered":"2024-05-06T00:26:46.716615+0000","last_clean":"2024-05-06T00:26:46.716615+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:26:46.716615+0000","last_undegraded":"2024-05-06T00:26:46.716615+0000","last_fullsized":"2024-05-06T00:26:46.716615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.733467+0000","last_change":"2024-05-06T00:26:45.733467+0000","last_active":"2024-05-06T00:26:45.733467+0000","last_peered":"2024-05-06T00:26:45.733467+0000","last_clean":"2024-05-06T00:26:45.733467+0000","last_became_active":"2024-05-06T00:26:43.703416+0000","last_became_peered":"2024-05-06T00:26:43.703416+0000","last_unstale":"2024-05-06T00:26:45.733467+0000","last_undegraded":"2024-05-06T00:26:45.733467+0000","last_fullsized":"2024-05-06T00:26:45.733467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:18:08.644329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.720211+0000","last_change":"2024-05-06T00:26:45.720211+0000","last_active":"2024-05-06T00:26:45.720211+0000","last_peered":"2024-05-06T00:26:45.720211+0000","last_clean":"2024-05-06T00:26:45.720211+0000","last_became_active":"2024-05-06T00:26:43.705235+0000","last_became_peered":"2024-05-06T00:26:43.705235+0000","last_unstale":"2024-05-06T00:26:45.720211+0000","last_undegraded":"2024-05-06T00:26:45.720211+0000","last_fullsized":"2024-05-06T00:26:45.720211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:30:21.228289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.728858+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:26:45.728858+0000","last_peered":"2024-05-06T00:26:45.728858+0000","last_clean":"2024-05-06T00:26:45.728858+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:26:45.728858+0000","last_undegraded":"2024-05-06T00:26:45.728858+0000","last_fullsized":"2024-05-06T00:26:45.728858+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716144+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:26:46.716144+0000","last_peered":"2024-05-06T00:26:46.716144+0000","last_clean":"2024-05-06T00:26:46.716144+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:26:46.716144+0000","last_undegraded":"2024-05-06T00:26:46.716144+0000","last_fullsized":"2024-05-06T00:26:46.716144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:23:56.454003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.715776+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:26:46.715776+0000","last_peered":"2024-05-06T00:26:46.715776+0000","last_clean":"2024-05-06T00:26:46.715776+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:26:46.715776+0000","last_undegraded":"2024-05-06T00:26:46.715776+0000","last_fullsized":"2024-05-06T00:26:46.715776+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.712444+0000","last_change":"2024-05-06T00:26:45.712444+0000","last_active":"2024-05-06T00:26:45.712444+0000","last_peered":"2024-05-06T00:26:45.712444+0000","last_clean":"2024-05-06T00:26:45.712444+0000","last_became_active":"2024-05-06T00:26:43.708328+0000","last_became_peered":"2024-05-06T00:26:43.708328+0000","last_unstale":"2024-05-06T00:26:45.712444+0000","last_undegraded":"2024-05-06T00:26:45.712444+0000","last_fullsized":"2024-05-06T00:26:45.712444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:03:19.422070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716856+0000","last_change":"2024-05-06T00:26:35.652664+0000","last_active":"2024-05-06T00:26:46.716856+0000","last_peered":"2024-05-06T00:26:46.716856+0000","last_clean":"2024-05-06T00:26:46.716856+0000","last_became_active":"2024-05-06T00:26:35.652017+0000","last_became_peered":"2024-05-06T00:26:35.652017+0000","last_unstale":"2024-05-06T00:26:46.716856+0000","last_undegraded":"2024-05-06T00:26:46.716856+0000","last_fullsized":"2024-05-06T00:26:46.716856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:34.935271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716432+0000","last_change":"2024-05-06T00:26:45.725541+0000","last_active":"2024-05-06T00:26:46.716432+0000","last_peered":"2024-05-06T00:26:46.716432+0000","last_clean":"2024-05-06T00:26:46.716432+0000","last_became_active":"2024-05-06T00:26:43.696938+0000","last_became_peered":"2024-05-06T00:26:43.696938+0000","last_unstale":"2024-05-06T00:26:46.716432+0000","last_undegraded":"2024-05-06T00:26:46.716432+0000","last_fullsized":"2024-05-06T00:26:46.716432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:34:00.196906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.729010+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:26:45.729010+0000","last_peered":"2024-05-06T00:26:45.729010+0000","last_clean":"2024-05-06T00:26:45.729010+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:26:45.729010+0000","last_undegraded":"2024-05-06T00:26:45.729010+0000","last_fullsized":"2024-05-06T00:26:45.729010+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":118,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716944+0000","last_change":"2024-05-06T00:26:30.239639+0000","last_active":"2024-05-06T00:26:46.716944+0000","last_peered":"2024-05-06T00:26:46.716944+0000","last_clean":"2024-05-06T00:26:46.716944+0000","last_became_active":"2024-05-06T00:26:30.237552+0000","last_became_peered":"2024-05-06T00:26:30.237552+0000","last_unstale":"2024-05-06T00:26:46.716944+0000","last_undegraded":"2024-05-06T00:26:46.716944+0000","last_fullsized":"2024-05-06T00:26:46.716944+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:44:57.011856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.715841+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:26:46.715841+0000","last_peered":"2024-05-06T00:26:46.715841+0000","last_clean":"2024-05-06T00:26:46.715841+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:26:46.715841+0000","last_undegraded":"2024-05-06T00:26:46.715841+0000","last_fullsized":"2024-05-06T00:26:46.715841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:01:07.651454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.721816+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:26:45.721816+0000","last_peered":"2024-05-06T00:26:45.721816+0000","last_clean":"2024-05-06T00:26:45.721816+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:26:45.721816+0000","last_undegraded":"2024-05-06T00:26:45.721816+0000","last_fullsized":"2024-05-06T00:26:45.721816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:28:27.864205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.717271+0000","last_change":"2024-05-06T00:26:45.710256+0000","last_active":"2024-05-06T00:26:46.717271+0000","last_peered":"2024-05-06T00:26:46.717271+0000","last_clean":"2024-05-06T00:26:46.717271+0000","last_became_active":"2024-05-06T00:26:43.710572+0000","last_became_peered":"2024-05-06T00:26:43.710572+0000","last_unstale":"2024-05-06T00:26:46.717271+0000","last_undegraded":"2024-05-06T00:26:46.717271+0000","last_fullsized":"2024-05-06T00:26:46.717271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:42:01.973064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.716539+0000","last_change":"2024-05-06T00:26:45.725952+0000","last_active":"2024-05-06T00:26:46.716539+0000","last_peered":"2024-05-06T00:26:46.716539+0000","last_clean":"2024-05-06T00:26:46.716539+0000","last_became_active":"2024-05-06T00:26:43.686980+0000","last_became_peered":"2024-05-06T00:26:43.686980+0000","last_unstale":"2024-05-06T00:26:46.716539+0000","last_undegraded":"2024-05-06T00:26:46.716539+0000","last_fullsized":"2024-05-06T00:26:46.716539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:20:42.390268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.732742+0000","last_change":"2024-05-06T00:26:45.732742+0000","last_active":"2024-05-06T00:26:45.732742+0000","last_peered":"2024-05-06T00:26:45.732742+0000","last_clean":"2024-05-06T00:26:45.732742+0000","last_became_active":"2024-05-06T00:26:43.700115+0000","last_became_peered":"2024-05-06T00:26:43.700115+0000","last_unstale":"2024-05-06T00:26:45.732742+0000","last_undegraded":"2024-05-06T00:26:45.732742+0000","last_fullsized":"2024-05-06T00:26:45.732742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:13:32.570808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.717590+0000","last_change":"2024-05-06T00:26:45.711599+0000","last_active":"2024-05-06T00:26:46.717590+0000","last_peered":"2024-05-06T00:26:46.717590+0000","last_clean":"2024-05-06T00:26:46.717590+0000","last_became_active":"2024-05-06T00:26:43.692092+0000","last_became_peered":"2024-05-06T00:26:43.692092+0000","last_unstale":"2024-05-06T00:26:46.717590+0000","last_undegraded":"2024-05-06T00:26:46.717590+0000","last_fullsized":"2024-05-06T00:26:46.717590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:12:42.492480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"21'3","reported_seq":39,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.884313+0000","last_change":"2024-05-06T00:26:45.734276+0000","last_active":"2024-05-06T00:26:45.884313+0000","last_peered":"2024-05-06T00:26:45.884313+0000","last_clean":"2024-05-06T00:26:45.884313+0000","last_became_active":"2024-05-06T00:26:43.711491+0000","last_became_peered":"2024-05-06T00:26:43.711491+0000","last_unstale":"2024-05-06T00:26:45.884313+0000","last_undegraded":"2024-05-06T00:26:45.884313+0000","last_fullsized":"2024-05-06T00:26:45.884313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:13:02.025180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":44,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.734362+0000","last_change":"2024-05-06T00:26:45.734362+0000","last_active":"2024-05-06T00:26:45.734362+0000","last_peered":"2024-05-06T00:26:45.734362+0000","last_clean":"2024-05-06T00:26:45.734362+0000","last_became_active":"2024-05-06T00:26:43.703972+0000","last_became_peered":"2024-05-06T00:26:43.703972+0000","last_unstale":"2024-05-06T00:26:45.734362+0000","last_undegraded":"2024-05-06T00:26:45.734362+0000","last_fullsized":"2024-05-06T00:26:45.734362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:25.270689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.732061+0000","last_change":"2024-05-06T00:26:45.732061+0000","last_active":"2024-05-06T00:26:45.732061+0000","last_peered":"2024-05-06T00:26:45.732061+0000","last_clean":"2024-05-06T00:26:45.732061+0000","last_became_active":"2024-05-06T00:26:43.699883+0000","last_became_peered":"2024-05-06T00:26:43.699883+0000","last_unstale":"2024-05-06T00:26:45.732061+0000","last_undegraded":"2024-05-06T00:26:45.732061+0000","last_fullsized":"2024-05-06T00:26:45.732061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:58:42.845428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.717103+0000","last_change":"2024-05-06T00:26:45.710994+0000","last_active":"2024-05-06T00:26:46.717103+0000","last_peered":"2024-05-06T00:26:46.717103+0000","last_clean":"2024-05-06T00:26:46.717103+0000","last_became_active":"2024-05-06T00:26:43.710745+0000","last_became_peered":"2024-05-06T00:26:43.710745+0000","last_unstale":"2024-05-06T00:26:46.717103+0000","last_undegraded":"2024-05-06T00:26:46.717103+0000","last_fullsized":"2024-05-06T00:26:46.717103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:03:22.081725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-06T00:26:45.713363+0000","last_change":"2024-05-06T00:26:45.713363+0000","last_active":"2024-05-06T00:26:45.713363+0000","last_peered":"2024-05-06T00:26:45.713363+0000","last_clean":"2024-05-06T00:26:45.713363+0000","last_became_active":"2024-05-06T00:26:43.692971+0000","last_became_peered":"2024-05-06T00:26:43.692971+0000","last_unstale":"2024-05-06T00:26:45.713363+0000","last_undegraded":"2024-05-06T00:26:45.713363+0000","last_fullsized":"2024-05-06T00:26:45.713363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:23:17.148371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-06T00:26:46.717417+0000","last_change":"2024-05-06T00:26:45.711743+0000","last_active":"2024-05-06T00:26:46.717417+0000","last_peered":"2024-05-06T00:26:46.717417+0000","last_clean":"2024-05-06T00:26:46.717417+0000","last_became_active":"2024-05-06T00:26:43.700214+0000","last_became_peered":"2024-05-06T00:26:43.700214+0000","last_unstale":"2024-05-06T00:26:46.717417+0000","last_undegraded":"2024-05-06T00:26:46.717417+0000","last_fullsized":"2024-05-06T00:26:46.717417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:43:47.649249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":65,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":64,"num_read_kb":51,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640265,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":73074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640263,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":73063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":668389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":78005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":78024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640263,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":663458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:26:49.482 INFO:tasks.thrashosds.thrasher:Setting 3.15 to [4, 1] 2024-05-06T00:26:49.482 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.15', '4', '1'] 2024-05-06T00:26:49.483 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.15 4 1 2024-05-06T00:26:49.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:26:49.495 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:26:49.497 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:49.497+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:49.507 INFO:teuthology.orchestra.run.smithi186.stdout:83 write 0xcdf9a00 thru 0xce07b98 (0xe199 bytes) 2024-05-06T00:26:49.513 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:26:49.513 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:26:49.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:26:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:26:49.554 INFO:teuthology.orchestra.run.smithi186.stdout:84 trunc from 0xce07c00 to 0x8cde00 2024-05-06T00:26:49.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:26:49.571 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:26:49.578 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:49.579 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:49.593 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:26:49.608 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:26:49.608 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:26:49.609 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:49.605+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:49.675 INFO:teuthology.orchestra.run.smithi186.stdout:87 write 0xaf09c00 thru 0xaf1891e (0xed1f bytes) 2024-05-06T00:26:49.677 INFO:teuthology.orchestra.run.smithi186.stdout:90 write 0x2cb9400 thru 0x2cbc1e5 (0x2de6 bytes) 2024-05-06T00:26:49.677 INFO:teuthology.orchestra.run.smithi186.stdout:92 punch from 0x7707200 to 0x770bc00, (0x4a00 bytes) 2024-05-06T00:26:49.678 INFO:teuthology.orchestra.run.smithi186.stdout:93 read 0x443c200 thru 0x443f61b (0x341c bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:94 write 0x312ba00 thru 0x313234e (0x694f bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:95 read 0x74c3800 thru 0x74c4cab (0x14ac bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:96 read 0xaa9ea00 thru 0xaaa1bed (0x31ee bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:98 write 0x3d1ec00 thru 0x3d2db35 (0xef36 bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:99 read 0x940c600 thru 0x941af35 (0xe936 bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:101 read 0x910ec00 thru 0x91121f5 (0x35f6 bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:102 read 0x14eac00 thru 0x14f1b38 (0x6f39 bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:104 read 0xac7f800 thru 0xac82a2c (0x322d bytes) 2024-05-06T00:26:49.680 INFO:teuthology.orchestra.run.smithi186.stdout:105 punch from 0x76dd200 to 0x76eae00, (0xdc00 bytes) 2024-05-06T00:26:49.698 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:49.697+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:49.704 INFO:teuthology.orchestra.run.smithi186.stdout:106 write 0xbe78800 thru 0xbe85c32 (0xd433 bytes) 2024-05-06T00:26:49.706 INFO:teuthology.orchestra.run.smithi186.stdout:107 read 0x66bb400 thru 0x66c3675 (0x8276 bytes) 2024-05-06T00:26:49.707 INFO:teuthology.orchestra.run.smithi186.stdout:108 read 0x6382e00 thru 0x638c9fa (0x9bfb bytes) 2024-05-06T00:26:49.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:49.730 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:26:49.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:26:49.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [] 2024-05-06T00:26:49.730 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:49.732 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:26:49.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:26:49.802 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:49.801+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:49.885 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:26:49.886 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:49.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:26:49.902 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:49.897+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:49.994 INFO:teuthology.orchestra.run.smithi186.stdout:110 write 0xed1b000 thru 0xed1c448 (0x1449 bytes) 2024-05-06T00:26:49.994 INFO:teuthology.orchestra.run.smithi186.stdout:111 write 0xc8d5e00 thru 0xc8e104d (0xb24e bytes) 2024-05-06T00:26:49.994 INFO:teuthology.orchestra.run.smithi186.stdout:112 write 0x9b8d400 thru 0x9b948b3 (0x74b4 bytes) 2024-05-06T00:26:49.994 INFO:teuthology.orchestra.run.smithi186.stdout:113 read 0x7aa8800 thru 0x7ab0506 (0x7d07 bytes) 2024-05-06T00:26:49.995 INFO:teuthology.orchestra.run.smithi186.stdout:114 punch from 0xa812600 to 0xa814e00, (0x2800 bytes) 2024-05-06T00:26:50.001 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:49.997+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:50.017 INFO:teuthology.orchestra.run.smithi186.stdout:118 trunc from 0xed1c600 to 0x353de00 2024-05-06T00:26:50.024 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:50.024 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:50.025 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:26:50.025 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:26:50.025 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:50.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:26:50.092 INFO:teuthology.orchestra.run.smithi186.stdout:120 write 0x71fc800 thru 0x7205d64 (0x9565 bytes) 2024-05-06T00:26:50.097 INFO:teuthology.orchestra.run.smithi186.stdout:122 punch from 0x3f6e400 to 0x3f73800, (0x5400 bytes) 2024-05-06T00:26:50.098 INFO:teuthology.orchestra.run.smithi186.stdout:123 punch from 0x3459e00 to 0x345d200, (0x3400 bytes) 2024-05-06T00:26:50.098 INFO:teuthology.orchestra.run.smithi186.stdout:124 punch from 0x35dde00 to 0x35eb000, (0xd200 bytes) 2024-05-06T00:26:50.098 INFO:teuthology.orchestra.run.smithi186.stdout:125 punch from 0x3c2be00 to 0x3c37c00, (0xbe00 bytes) 2024-05-06T00:26:50.098 INFO:teuthology.orchestra.run.smithi186.stdout:126 read 0x171e000 thru 0x1721a09 (0x3a0a bytes) 2024-05-06T00:26:50.101 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:50.101+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:50.108 INFO:teuthology.orchestra.run.smithi186.stdout:127 trunc from 0x7205e00 to 0x45c2a00 2024-05-06T00:26:50.132 INFO:teuthology.orchestra.run.smithi186.stdout:131 write 0x2d1ac00 thru 0x2d1dd54 (0x3155 bytes) 2024-05-06T00:26:50.135 INFO:teuthology.orchestra.run.smithi186.stdout:132 read 0x204c400 thru 0x2055b26 (0x9727 bytes) 2024-05-06T00:26:50.136 INFO:teuthology.orchestra.run.smithi186.stdout:133 read 0x14e3a00 thru 0x14f1089 (0xd68a bytes) 2024-05-06T00:26:50.137 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0x9bbe400 2024-05-06T00:26:50.154 INFO:teuthology.orchestra.run.smithi186.stdout:135 trunc from 0x45c2a00 to 0x9bbe400 2024-05-06T00:26:50.173 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:50.173 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:26:50.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:26:50.173 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:26:50.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.174 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.174 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:26:50.174 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.565541562, 2024-05-06T00:26:50.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:26:50.175 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.175 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.175 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.176 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:26:50.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.177 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:26:50.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:26:50.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:26:50.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:26:50.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:23 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.277392+0000", 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.890549176, 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088550460000000001, 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.184 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 23 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277392+0000", 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277392+0000", 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277394+0000", 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.12e-06 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277596+0000", 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020261500000000001 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277619+0000", 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3223000000000001e-05 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277637+0000", 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.717e-05 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.280784+0000", 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031470999999999999 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.280907+0000", 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012304199999999999 2024-05-06T00:26:50.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.281166+0000", 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025873500000000001 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.285379+0000", 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042136669999999999 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.286115+0000", 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00073536800000000002 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.286189+0000", 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4402999999999999e-05 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.286247+0000", 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7602999999999997e-05 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:24 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.277667+0000", 2024-05-06T00:26:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.890273334, 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.009192492, 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 24 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277667+0000", 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277667+0000", 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277669+0000", 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6330000000000001e-06 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277870+0000", 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020082499999999999 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277895+0000", 2024-05-06T00:26:50.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4803e-05 2024-05-06T00:26:50.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277933+0000", 2024-05-06T00:26:50.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8059999999999998e-05 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.282477+0000", 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045444049999999996 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.282625+0000", 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147548 2024-05-06T00:26:50.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.282843+0000", 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218352 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.285386+0000", 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002543324 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.286737+0000", 2024-05-06T00:26:50.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00135111 2024-05-06T00:26:50.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.286805+0000", 2024-05-06T00:26:50.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7860000000000004e-05 2024-05-06T00:26:50.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.286860+0000", 2024-05-06T00:26:50.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4571999999999999e-05 2024-05-06T00:26:50.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.203 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.204 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.205 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.277972+0000", 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.889968393, 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091725460000000002, 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 25 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.206 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277972+0000", 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277972+0000", 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277974+0000", 2024-05-06T00:26:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6160000000000001e-06 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.277987+0000", 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3427e-05 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.278013+0000", 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5372999999999999e-05 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.278031+0000", 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8615999999999999e-05 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.283560+0000", 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055285739999999996 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.283821+0000", 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026118499999999999 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.284198+0000", 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00037729700000000002 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.285398+0000", 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011995739999999999 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.286949+0000", 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015509020000000001 2024-05-06T00:26:50.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.287039+0000", 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9685999999999995e-05 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.287145+0000", 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106296 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.380677+0000", 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.787264096000001, 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085763769999999996, 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 42 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380677+0000", 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380677+0000", 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380678+0000", 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6130000000000001e-06 2024-05-06T00:26:50.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380689+0000", 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1029999999999999e-05 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380712+0000", 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2782999999999999e-05 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380727+0000", 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.539e-05 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.384308+0000", 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003580546 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.384428+0000", 2024-05-06T00:26:50.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120239 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.384630+0000", 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020195200000000001 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.388542+0000", 2024-05-06T00:26:50.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039115790000000001 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389082+0000", 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054078000000000004 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389178+0000", 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5451999999999993e-05 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389253+0000", 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5012999999999995e-05 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.380841+0000", 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.787099517000001, 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098001270000000005, 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.218 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 43 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380841+0000", 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380841+0000", 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380843+0000", 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.68e-06 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380855+0000", 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.208e-05 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380878+0000", 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2616000000000002e-05 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.380894+0000", 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.605e-05 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.385945+0000", 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050511610000000002 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.386081+0000", 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013590499999999999 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.386310+0000", 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022978199999999999 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389404+0000", 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003093153 2024-05-06T00:26:50.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390496+0000", 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010923510000000001 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390581+0000", 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5463000000000005e-05 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390641+0000", 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9886000000000002e-05 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381007+0000", 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.786934099, 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010854519, 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.224 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 44 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381007+0000", 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381007+0000", 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381008+0000", 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8139999999999999e-06 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381128+0000", 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011914200000000001 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381151+0000", 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3413000000000001e-05 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381171+0000", 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0373e-05 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.387069+0000", 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005897312 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.387247+0000", 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017819799999999999 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.387464+0000", 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021704800000000001 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391710+0000", 2024-05-06T00:26:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042457409999999999 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391734+0000", 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4283000000000001e-05 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391804+0000", 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9795999999999994e-05 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391861+0000", 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7398999999999997e-05 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.786711777000001, 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012319788, 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.230 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381231+0000", 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.624e-06 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381243+0000", 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2476e-05 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381263+0000", 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-05 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381278+0000", 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5183e-05 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389490+0000", 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082113770000000006 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389614+0000", 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012416900000000001 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389813+0000", 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199321 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391953+0000", 2024-05-06T00:26:50.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021402869999999998 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393301+0000", 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001347729 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393438+0000", 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136666 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393549+0000", 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110856 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.786635973999999, 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014641942999999999, 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381306+0000", 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.403e-06 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381316+0000", 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381341+0000", 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4372999999999999e-05 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381356+0000", 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5207e-05 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390708+0000", 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093519799999999993 2024-05-06T00:26:50.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390856+0000", 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839200000000001 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391064+0000", 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000207302 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394636+0000", 2024-05-06T00:26:50.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003572563 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395813+0000", 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011765910000000001 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395885+0000", 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2601999999999994e-05 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395947+0000", 2024-05-06T00:26:50.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1183e-05 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.786559298, 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017147241000000001, 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.242 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 47 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:26:50.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381383+0000", 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6199999999999999e-06 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381504+0000", 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120709 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381525+0000", 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1705999999999999e-05 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381541+0000", 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5367e-05 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392051+0000", 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010510442 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392177+0000", 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012575600000000001 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392387+0000", 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020954799999999999 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394744+0000", 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023578319999999998 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398377+0000", 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036325189999999999 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398462+0000", 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4668999999999999e-05 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398529+0000", 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7073e-05 2024-05-06T00:26:50.247 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.786359282999999, 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017264267, 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.248 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 48 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381583+0000", 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7069999999999999e-06 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381594+0000", 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1365999999999999e-05 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381618+0000", 2024-05-06T00:26:50.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4014000000000001e-05 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381639+0000", 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0616e-05 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393664+0000", 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012025179 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393804+0000", 2024-05-06T00:26:50.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014017599999999999 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393998+0000", 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019390499999999999 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398573+0000", 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045742789999999997 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398683+0000", 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110136 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398786+0000", 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103259 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398846+0000", 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9630000000000003e-05 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.253 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.786268020000001, 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018281285000000001, 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 49 2024-05-06T00:26:50.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381674+0000", 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.559e-06 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381685+0000", 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1049999999999999e-05 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381705+0000", 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0227000000000001e-05 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381728+0000", 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2467e-05 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394829+0000", 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013101358 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394953+0000", 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012340499999999999 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395145+0000", 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019252900000000001 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399714+0000", 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045683620000000003 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399788+0000", 2024-05-06T00:26:50.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4719000000000001e-05 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399871+0000", 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3092999999999999e-05 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399954+0000", 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2515999999999994e-05 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.786178340999999, 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020991408, 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 50 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.260 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381764+0000", 2024-05-06T00:26:50.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381894+0000", 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012972600000000001 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381927+0000", 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3399000000000003e-05 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381938+0000", 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0677e-05 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396032+0000", 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014094257000000001 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396199+0000", 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016722899999999999 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396415+0000", 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021597200000000001 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.400244+0000", 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038293009999999998 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402502+0000", 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002257972 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402645+0000", 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014225199999999999 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402754+0000", 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010917300000000001 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.265 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.785980966, 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021178922999999999, 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 51 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381961+0000", 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381968+0000", 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.007e-06 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381979+0000", 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1197e-05 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381987+0000", 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9629999999999998e-06 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397517+0000", 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015529589999999999 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397635+0000", 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011846899999999999 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397830+0000", 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019498200000000001 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402867+0000", 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050367240000000002 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403002+0000", 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134609 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403082+0000", 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0666000000000003e-05 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403139+0000", 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6462999999999998e-05 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:26:50.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.785940209, 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021264655, 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:26:50.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382001+0000", 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7300000000000005e-07 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382007+0000", 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000004e-06 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382017+0000", 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9799999999999993e-06 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382029+0000", 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1420000000000001e-05 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398910+0000", 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016881075999999998 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399069+0000", 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015880199999999999 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399327+0000", 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025800200000000001 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402970+0000", 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003643725 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403155+0000", 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018459100000000001 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403221+0000", 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6106999999999996e-05 2024-05-06T00:26:50.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403265+0000", 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4135999999999999e-05 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585348+0000", 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.582593047, 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.009936828, 2024-05-06T00:26:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585348+0000", 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585348+0000", 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585349+0000", 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.043e-06 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585444+0000", 2024-05-06T00:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5558999999999998e-05 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585455+0000", 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1211e-05 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585466+0000", 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0224000000000001e-05 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.591129+0000", 2024-05-06T00:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056629230000000003 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.591221+0000", 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2254000000000001e-05 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.591420+0000", 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199438 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593616+0000", 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021954940000000001 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595192+0000", 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015766 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595231+0000", 2024-05-06T00:26:50.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498000000000003e-05 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595284+0000", 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3584000000000001e-05 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585484+0000", 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.582456978, 2024-05-06T00:26:50.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099164160000000008, 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 73 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585484+0000", 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585484+0000", 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585485+0000", 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4499999999999995e-07 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585496+0000", 2024-05-06T00:26:50.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0875e-05 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585525+0000", 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9179e-05 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585534+0000", 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9749999999999996e-06 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.592088+0000", 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065547180000000002 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.592147+0000", 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8461000000000002e-05 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.592242+0000", 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4728000000000005e-05 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595150+0000", 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029086530000000002 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595328+0000", 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017762299999999999 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595367+0000", 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9431999999999998e-05 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595400+0000", 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2827000000000003e-05 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585554+0000", 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.582387009000001, 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012126827999999999, 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.289 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585554+0000", 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585554+0000", 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585555+0000", 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4600000000000003e-07 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585560+0000", 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585571+0000", 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0716999999999999e-05 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585579+0000", 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5850000000000002e-06 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.592955+0000", 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073760739999999998 2024-05-06T00:26:50.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593080+0000", 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012530000000000001 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593283+0000", 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020279599999999999 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595409+0000", 2024-05-06T00:26:50.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00212664 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.597599+0000", 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021895170000000002 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.597635+0000", 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5744000000000002e-05 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.597680+0000", 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5872999999999999e-05 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.582342891, 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012456859000000001, 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.295 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:26:50.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585599+0000", 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5899999999999995e-07 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585692+0000", 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3398000000000003e-05 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585704+0000", 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2318e-05 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585713+0000", 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7649999999999999e-06 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593668+0000", 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079544980000000008 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593723+0000", 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5914e-05 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593815+0000", 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1583999999999996e-05 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595554+0000", 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017386190000000001 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.597968+0000", 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024139500000000002 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598008+0000", 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9962000000000001e-05 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598055+0000", 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7092000000000002e-05 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.582210632999999, 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013009979, 2024-05-06T00:26:50.301 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 76 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585731+0000", 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0999999999999997e-07 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585738+0000", 2024-05-06T00:26:50.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6059999999999999e-06 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585752+0000", 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4248e-05 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585762+0000", 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8660000000000001e-06 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594480+0000", 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087185100000000005 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594616+0000", 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135375 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594806+0000", 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019038099999999999 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595986+0000", 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011804039999999999 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598676+0000", 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026891889999999998 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598710+0000", 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4471999999999998e-05 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598740+0000", 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0018e-05 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.307 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:50.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:26:50.310 INFO:teuthology.orchestra.run.smithi186.stdout:139 read 0x6f04200 thru 0x6f0fd06 (0xbb07 bytes) 2024-05-06T00:26:50.310 INFO:teuthology.orchestra.run.smithi186.stdout:140 read 0xa26e00 thru 0xa2bdd9 (0x4fda bytes) 2024-05-06T00:26:50.310 INFO:teuthology.orchestra.run.smithi186.stdout:142 read 0x399c800 thru 0x39a1645 (0x4e46 bytes) 2024-05-06T00:26:50.310 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0xc3c9400 2024-05-06T00:26:50.310 INFO:teuthology.orchestra.run.smithi186.stdout:149 trunc from 0x9bbe400 to 0xc3c9400 2024-05-06T00:26:50.310 INFO:teuthology.orchestra.run.smithi186.stdout:151 write 0x2573e00 thru 0x257d806 (0x9a07 bytes) 2024-05-06T00:26:50.311 INFO:teuthology.orchestra.run.smithi186.stdout:152 read 0x5d8b800 thru 0x5d8c636 (0xe37 bytes) 2024-05-06T00:26:50.311 INFO:teuthology.orchestra.run.smithi186.stdout:153 write 0xa421800 thru 0xa429693 (0x7e94 bytes) 2024-05-06T00:26:50.311 INFO:teuthology.orchestra.run.smithi186.stdout:154 clone 2 order 25 su 16777216 sc 12 2024-05-06T00:26:50.311 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:50.305+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:50.409 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:50.409+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:50.484 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:50.484 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:50.484 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:26:50.485 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:50.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:26:50.510 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:50.509+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:50.611 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:50.609+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:50.665 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:50.665 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:50.665 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:26:50.665 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:26:50.666 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:50.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:26:50.724 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:50.725+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:set 3.15 pg_upmap mapping to [4,1] 2024-05-06T00:26:50.806 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:26:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:26:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:26:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:9 3.1a 3:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:26:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0899335399999996, 2024-05-06T00:26:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084089950000000007, 2024-05-06T00:26:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 9 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715203+0000", 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715219+0000", 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6577000000000001e-05 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715254+0000", 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5308999999999997e-05 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715279+0000", 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4902999999999999e-05 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715445+0000", 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016536900000000001 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715456+0000", 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1037e-05 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.719765+0000", 2024-05-06T00:26:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043095060000000003 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.720002+0000", 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000236596 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.720369+0000", 2024-05-06T00:26:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036678999999999998 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.722404+0000", 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002035121 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723446+0000", 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010426249999999999 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723534+0000", 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7423000000000006e-05 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723607+0000", 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3195999999999995e-05 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:467 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'38, mlcod=23'36)", 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.591645+0000", 2024-05-06T00:26:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.2134861400000001, 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019792310000000001, 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.591645+0000", 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.591645+0000", 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.591647+0000", 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.846e-06 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.591659+0000", 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1963999999999999e-05 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.591687+0000", 2024-05-06T00:26:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8005999999999999e-05 2024-05-06T00:26:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.591702+0000", 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4634000000000001e-05 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.591817+0000", 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115442 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.591875+0000", 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7383000000000003e-05 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.593522+0000", 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001647667 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.593625+0000", 2024-05-06T00:26:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102289 2024-05-06T00:26:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:469 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'39, mlcod=23'37)", 2024-05-06T00:26:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.597340+0000", 2024-05-06T00:26:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.2077919500000001, 2024-05-06T00:26:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021042830000000002, 2024-05-06T00:26:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.597340+0000", 2024-05-06T00:26:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.597340+0000", 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.597341+0000", 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6729999999999999e-06 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.597356+0000", 2024-05-06T00:26:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.489e-05 2024-05-06T00:26:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.597383+0000", 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6523e-05 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.597395+0000", 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2676e-05 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.597539+0000", 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014348900000000001 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.597598+0000", 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9596999999999999e-05 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.599342+0000", 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017437430000000001 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.599444+0000", 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101692 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:470 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'40, mlcod=23'38)", 2024-05-06T00:26:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.600396+0000", 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.2047350610000001, 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022031970000000001, 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.600396+0000", 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.600396+0000", 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.600398+0000", 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.600407+0000", 2024-05-06T00:26:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8969999999999997e-06 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.600429+0000", 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1449000000000001e-05 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.600441+0000", 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.223e-05 2024-05-06T00:26:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.600557+0000", 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116119 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.600619+0000", 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1470000000000006e-05 2024-05-06T00:26:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.602484+0000", 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018653649999999999 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.602600+0000", 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011576 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:477 3.1a 3:5d209d14:::rbd_data.110c17d674e9.0000000000000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.619974+0000", 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.1851579219999999, 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024439150000000001, 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 477 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.619974+0000", 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.619974+0000", 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.619976+0000", 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6570000000000001e-06 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.619987+0000", 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1080000000000001e-05 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.620018+0000", 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0985999999999998e-05 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.620033+0000", 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4929999999999999e-05 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.620181+0000", 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147615 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.620421+0000", 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024054499999999999 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.620758+0000", 2024-05-06T00:26:50.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033706099999999999 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.622126+0000", 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001367666 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.622161+0000", 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5330000000000002e-05 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.622299+0000", 2024-05-06T00:26:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013762900000000001 2024-05-06T00:26:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.622418+0000", 2024-05-06T00:26:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118416 2024-05-06T00:26:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:545 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'48, mlcod=23'46)", 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.1238526449999999, 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004580738, 2024-05-06T00:26:50.842 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:26:50.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681281+0000", 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681293+0000", 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681321+0000", 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8073e-05 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681342+0000", 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1192999999999999e-05 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681467+0000", 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000124976 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681503+0000", 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6207000000000001e-05 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685774+0000", 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042710869999999998 2024-05-06T00:26:50.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685860+0000", 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5228999999999993e-05 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:546 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'49, mlcod=23'46)", 2024-05-06T00:26:50.847 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.681820+0000", 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.1233110959999999, 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040870250000000002, 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681820+0000", 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681820+0000", 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681821+0000", 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9000000000000005e-07 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681867+0000", 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5370000000000001e-05 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681898+0000", 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1216000000000001e-05 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681914+0000", 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6157000000000001e-05 2024-05-06T00:26:50.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682064+0000", 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014957499999999999 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682099+0000", 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5309999999999999e-05 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685867+0000", 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037681070000000001 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685907+0000", 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0299999999999997e-05 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:547 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'50, mlcod=23'46)", 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.682463+0000", 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.122668861, 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034892159999999998, 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682463+0000", 2024-05-06T00:26:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682463+0000", 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682464+0000", 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682477+0000", 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3186999999999999e-05 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682500+0000", 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3173e-05 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682515+0000", 2024-05-06T00:26:50.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4552999999999999e-05 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682702+0000", 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018694600000000001 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.682760+0000", 2024-05-06T00:26:50.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8482e-05 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685909+0000", 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031488929999999998 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685952+0000", 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2589000000000002e-05 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:550 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'51, mlcod=23'46)", 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.683549+0000", 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.1215826959999999, 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002469714, 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.858 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.683549+0000", 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.683549+0000", 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.683550+0000", 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.46e-06 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.683563+0000", 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2967e-05 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.683591+0000", 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7396e-05 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.683606+0000", 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5637e-05 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.683724+0000", 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117549 2024-05-06T00:26:50.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.683780+0000", 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6076000000000003e-05 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685979+0000", 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021987629999999998 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686019+0000", 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9866000000000001e-05 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:551 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'52, mlcod=23'46)", 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.684188+0000", 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.120943668, 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002590097, 2024-05-06T00:26:50.863 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.684188+0000", 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.684188+0000", 2024-05-06T00:26:50.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.684189+0000", 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2270000000000001e-06 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.684201+0000", 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1909999999999999e-05 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.684247+0000", 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6156000000000003e-05 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.684282+0000", 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4653000000000002e-05 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.684706+0000", 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042394999999999998 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.684814+0000", 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108439 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686697+0000", 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018825090000000001 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686778+0000", 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1253000000000003e-05 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:554 3.1b e23/18 3:da0a6ebf:::rbd_data.110c17d674e9.0000000000000003:head v 23'53, mlcod=23'50)", 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.686678+0000", 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.1184535229999999, 2024-05-06T00:26:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018681450000000001, 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686678+0000", 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686678+0000", 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686679+0000", 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.37e-06 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686752+0000", 2024-05-06T00:26:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3091999999999997e-05 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686778+0000", 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5950000000000001e-05 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686795+0000", 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6872999999999999e-05 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686887+0000", 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2108999999999995e-05 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.686920+0000", 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2193999999999997e-05 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688452+0000", 2024-05-06T00:26:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015319109999999999 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688546+0000", 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4646000000000002e-05 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:559 3.1a 3:5d209d14:::rbd_data.110c17d674e9.0000000000000007:head [set-alloc-hint object_size 16777216 write_size 16777216,write 5365760~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.11756597, 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034544290000000002, 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 559 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687569+0000", 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9799999999999998e-06 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687724+0000", 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155422 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687762+0000", 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7935999999999997e-05 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687784+0000", 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2589999999999999e-05 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687940+0000", 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155439 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688235+0000", 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029508699999999998 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688641+0000", 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040601400000000001 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690825+0000", 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021841159999999998 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690859+0000", 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4173e-05 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690950+0000", 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0610000000000002e-05 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691020+0000", 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0061999999999994e-05 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:632 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.067406004, 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073037140000000002, 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.880 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 632 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737728+0000", 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6699999999999998e-06 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737738+0000", 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0263e-05 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737764+0000", 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5584000000000001e-05 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737781+0000", 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6849000000000001e-05 2024-05-06T00:26:50.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737901+0000", 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012037299999999999 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737912+0000", 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0343000000000001e-05 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.744738+0000", 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068260940000000004 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.744863+0000", 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125139 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745029+0000", 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016639899999999999 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:342 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.065158048, 2024-05-06T00:26:50.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053952979999999998, 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 342 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739976+0000", 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.88e-06 2024-05-06T00:26:50.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739988+0000", 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1440000000000001e-05 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740019+0000", 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0942999999999998e-05 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740034+0000", 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4766000000000001e-05 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740219+0000", 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185069 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740229+0000", 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0890000000000001e-05 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745083+0000", 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048533760000000004 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745208+0000", 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012539499999999999 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745369+0000", 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016053899999999999 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:711 3.1a 3:5d209d14:::rbd_data.110c17d674e9.0000000000000007:head [truncate 4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.045179+0000", 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.75995230899999999, 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020786619999999998, 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.892 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 711 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045179+0000", 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045179+0000", 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045181+0000", 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9120000000000001e-06 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045191+0000", 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0173999999999999e-05 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045209+0000", 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7394e-05 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045217+0000", 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6859999999999995e-06 2024-05-06T00:26:50.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045274+0000", 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6953999999999997e-05 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045341+0000", 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7084999999999999e-05 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.045442+0000", 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100536 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.046042+0000", 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00059982600000000003 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.047168+0000", 2024-05-06T00:26:50.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001125841 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.047213+0000", 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4926000000000003e-05 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.047258+0000", 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5327999999999997e-05 2024-05-06T00:26:50.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:772 3.1a 3:5d209d14:::rbd_data.110c17d674e9.0000000000000007:head [write 7791616~21504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.100972+0000", 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.70415972900000001, 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030315229999999999, 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.100972+0000", 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.100972+0000", 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.100973+0000", 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.232e-06 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.100994+0000", 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1302999999999999e-05 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.101010+0000", 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6126000000000001e-05 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.101019+0000", 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0499999999999992e-06 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.101076+0000", 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7024000000000002e-05 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.101147+0000", 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-05-06T00:26:50.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.101269+0000", 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012162700000000001 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.102292+0000", 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001022954 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.103902+0000", 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001610034 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.103956+0000", 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4552000000000002e-05 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104003+0000", 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6842000000000003e-05 2024-05-06T00:26:50.905 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:778 3.1a 3:5d209d14:::rbd_data.110c17d674e9.0000000000000007:head [write 4374016~48640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.104333+0000", 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.70079894399999998, 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002231547, 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.906 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 778 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104333+0000", 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104333+0000", 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104333+0000", 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3099999999999996e-07 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104365+0000", 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1146000000000003e-05 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104374+0000", 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4420000000000004e-06 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104378+0000", 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4660000000000001e-06 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104419+0000", 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0524e-05 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104480+0000", 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0795999999999999e-05 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.104598+0000", 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118114 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.105305+0000", 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070660700000000005 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.106419+0000", 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011141370000000001 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.106473+0000", 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3925999999999998e-05 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.106564+0000", 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1558000000000007e-05 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:852 3.1b e24/18 3:da0a6ebf:::rbd_data.110c17d674e9.0000000000000003:head v 24'68, mlcod=24'66)", 2024-05-06T00:26:50.913 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.64879800200000004, 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041048589999999998, 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156335+0000", 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0580000000000001e-06 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156365+0000", 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0719999999999997e-05 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156384+0000", 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8715e-05 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156395+0000", 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0597999999999999e-05 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156495+0000", 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100055 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156527+0000", 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2458e-05 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.160390+0000", 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038625040000000001 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.160438+0000", 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8751000000000001e-05 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.920 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:944 3.1b e24/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 24'71, mlcod=24'69)", 2024-05-06T00:26:50.920 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.225575+0000", 2024-05-06T00:26:50.920 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.57955634599999994, 2024-05-06T00:26:50.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002057578, 2024-05-06T00:26:50.920 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.920 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.920 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.225575+0000", 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.225575+0000", 2024-05-06T00:26:50.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.225576+0000", 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-06 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.225585+0000", 2024-05-06T00:26:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9639999999999992e-06 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.225602+0000", 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6633000000000002e-05 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.225612+0000", 2024-05-06T00:26:50.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7559999999999994e-06 2024-05-06T00:26:50.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.225691+0000", 2024-05-06T00:26:50.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8949999999999995e-05 2024-05-06T00:26:50.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.225728+0000", 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7659e-05 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.227600+0000", 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001871663 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.227633+0000", 2024-05-06T00:26:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2756999999999998e-05 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:948 3.a e24/18 3:542123e8:::rbd_data.110c17d674e9.0000000000000009:head v 24'5, mlcod=23'3)", 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.228894+0000", 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.57623792500000004, 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021536039999999999, 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.228894+0000", 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.228894+0000", 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.228895+0000", 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8169999999999999e-06 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.228955+0000", 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0053000000000003e-05 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.228983+0000", 2024-05-06T00:26:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7699e-05 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.228996+0000", 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2507000000000001e-05 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.229123+0000", 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127522 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.229183+0000", 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9947000000000001e-05 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.230953+0000", 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017699160000000001 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.231047+0000", 2024-05-06T00:26:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4142999999999996e-05 2024-05-06T00:26:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:50.931 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:50.932 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone1 from 0xc3c9400 (overlap 0x8cde00) to 0x466e00 2024-05-06T00:26:50.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:26:50.933 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:50.929+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:51.033 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:51.029+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:51.105 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:26:51.105 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:51.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:26:51.164 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:51.165+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:51.236 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:51.233+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:51.243 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:51.243 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:51.243 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:26:51.243 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:26:51.243 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:51.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:26:51.352 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:51.353+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:10 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7113323310000004, 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074329959999999999, 2024-05-06T00:26:51.438 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 10 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725266+0000", 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8830000000000002e-06 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725276+0000", 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.042e-05 2024-05-06T00:26:51.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725297+0000", 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0927000000000001e-05 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725314+0000", 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7266e-05 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725442+0000", 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012823899999999999 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725452+0000", 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.944e-06 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729257+0000", 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038050530000000001 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729483+0000", 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022601199999999999 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729822+0000", 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033856400000000002 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.731316+0000", 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014941209999999999 2024-05-06T00:26:51.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732407+0000", 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010905820000000001 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732553+0000", 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014660899999999999 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732696+0000", 2024-05-06T00:26:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014237599999999999 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:12 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.737020+0000", 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6995752379999995, 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003051893, 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 12 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737020+0000", 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737020+0000", 2024-05-06T00:26:51.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737022+0000", 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5459999999999998e-06 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737036+0000", 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3886999999999999e-05 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737065+0000", 2024-05-06T00:26:51.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9105999999999999e-05 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737081+0000", 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5307000000000001e-05 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737219+0000", 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013803200000000001 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737454+0000", 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023531499999999999 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737835+0000", 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038118700000000001 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.739250+0000", 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014146919999999999 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.739911+0000", 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066157200000000003 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.740002+0000", 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0172000000000003e-05 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.740072+0000", 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0077e-05 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.450 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.451 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:25 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 139651801167792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:51.451 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.765469+0000", 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6711264879999996, 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026801590000000001, 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 25 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765469+0000", 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765469+0000", 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765471+0000", 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2730000000000001e-06 2024-05-06T00:26:51.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765481+0000", 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0190000000000001e-05 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765512+0000", 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1013999999999998e-05 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765530+0000", 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8318999999999999e-05 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765689+0000", 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015869500000000001 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765825+0000", 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136456 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.766057+0000", 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023154499999999999 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.767205+0000", 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011481149999999999 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.767887+0000", 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068138699999999999 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.767979+0000", 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2289999999999999e-05 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.768149+0000", 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000169875 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4467.0:7 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [call rbd.get_size in=11b,call rbd.get_features in=15b,call rbd.get_flags in=12b,call,call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.849826+0000", 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.5867689379999996, 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001514121, 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4467", 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3050489859", 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 7 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.849826+0000", 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.849826+0000", 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.849828+0000", 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5629999999999999e-06 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.849837+0000", 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4299999999999995e-06 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.849859+0000", 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1543e-05 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.849873+0000", 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.419e-05 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.850030+0000", 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015731499999999999 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.850292+0000", 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026233499999999999 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.851340+0000", 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010477450000000001 2024-05-06T00:26:51.461 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4467.0:12 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 139796487965376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.857907+0000", 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.5786876339999996, 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029019380000000002, 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4467", 2024-05-06T00:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3050489859", 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 12 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857907+0000", 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857907+0000", 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857909+0000", 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6330000000000001e-06 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857918+0000", 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4399999999999994e-06 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857941+0000", 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2903000000000001e-05 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857954+0000", 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2320000000000001e-05 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.858069+0000", 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011578600000000001 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.858296+0000", 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022668800000000001 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.858770+0000", 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047380399999999998 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.860231+0000", 2024-05-06T00:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014605849999999999 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.860629+0000", 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039847999999999999 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.860710+0000", 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0646e-05 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.860809+0000", 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9653000000000003e-05 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:556 3.17 e23/18 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head v 23'22, mlcod=23'20)", 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.7494303709999999, 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039476930000000004, 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687167+0000", 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9269999999999999e-06 2024-05-06T00:26:51.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687457+0000", 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029068099999999999 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687495+0000", 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7870000000000002e-05 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687514+0000", 2024-05-06T00:26:51.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8419000000000001e-05 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687650+0000", 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136966 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687692+0000", 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1029999999999998e-05 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690995+0000", 2024-05-06T00:26:51.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033034140000000002 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691112+0000", 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117386 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:557 3.17 e23/18 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head v 23'23, mlcod=23'20)", 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.749029084, 2024-05-06T00:26:51.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039228129999999998, 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:51.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687567+0000", 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2869999999999999e-06 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687619+0000", 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2119e-05 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687640+0000", 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1123000000000001e-05 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687656+0000", 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.523e-05 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688969+0000", 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00131369 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.689027+0000", 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000001e-05 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691373+0000", 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023457819999999998 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691489+0000", 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116042 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:558 3.17 e23/18 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head v 23'24, mlcod=23'20)", 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.7483720489999999, 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037091110000000002, 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688225+0000", 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4940000000000002e-06 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688325+0000", 2024-05-06T00:26:51.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9679000000000006e-05 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688356+0000", 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1313000000000003e-05 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688377+0000", 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0970000000000001e-05 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690069+0000", 2024-05-06T00:26:51.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016917360000000001 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690106+0000", 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6912999999999997e-05 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691873+0000", 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017671200000000001 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691932+0000", 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8885999999999998e-05 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:690 3.2 e24/18 3:405a3ab9:::rbd_data.110c17d674e9.000000000000000a:head v 24'3, mlcod=23'1)", 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.004869+0000", 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.431725505, 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002965526, 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.004869+0000", 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.004869+0000", 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.004870+0000", 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7700000000000001e-07 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005118+0000", 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024812499999999997 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005142+0000", 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3697999999999999e-05 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005154+0000", 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.218e-05 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005238+0000", 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4537999999999996e-05 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005272+0000", 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3408000000000003e-05 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.007786+0000", 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025146259999999998 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.007835+0000", 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:694 3.2 e24/18 3:405a3ab9:::rbd_data.110c17d674e9.000000000000000a:head v 24'5, mlcod=24'3)", 2024-05-06T00:26:51.485 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.031413+0000", 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.405181759, 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015943260000000001, 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031413+0000", 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031413+0000", 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031415+0000", 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6449999999999999e-06 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031426+0000", 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1260000000000001e-05 2024-05-06T00:26:51.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031446+0000", 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9942999999999999e-05 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031456+0000", 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8209999999999997e-06 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031584+0000", 2024-05-06T00:26:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012832100000000001 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031644+0000", 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0123e-05 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.032933+0000", 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012884859999999999 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.033008+0000", 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4727000000000005e-05 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:707 3.e 3:74163812:::rbd_data.110c17d674e9.0000000000000002:head [stat,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.040217+0000", 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.3963780720000001, 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002539829, 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.490 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 707 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040217+0000", 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040217+0000", 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040220+0000", 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5370000000000002e-06 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040234+0000", 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3439e-05 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040269+0000", 2024-05-06T00:26:51.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5240000000000001e-05 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040285+0000", 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6317e-05 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040416+0000", 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130359 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040569+0000", 2024-05-06T00:26:51.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015351199999999999 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040821+0000", 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025124199999999999 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.041812+0000", 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00099124499999999997 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.042686+0000", 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087427800000000003 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.042723+0000", 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7237000000000003e-05 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.042757+0000", 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3423000000000002e-05 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:426 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.781779+0000", 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.65481565200000003, 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029378160000000002, 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 426 2024-05-06T00:26:51.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781779+0000", 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781779+0000", 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781783+0000", 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2669999999999998e-06 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781798+0000", 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5173e-05 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781832+0000", 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4341999999999997e-05 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781849+0000", 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.658e-05 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781948+0000", 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9444000000000005e-05 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781959+0000", 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0427e-05 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.784464+0000", 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025050789999999999 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.784562+0000", 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8096000000000004e-05 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.784717+0000", 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155408 2024-05-06T00:26:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:446 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [call rbd.get_size in=11b,call rbd.get_features in=15b,call rbd.get_flags in=12b,call,call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.827563+0000", 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.60903184899999996, 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001401062, 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 446 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.827563+0000", 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.827563+0000", 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.827565+0000", 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1160000000000002e-06 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.827576+0000", 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.093e-05 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.827605+0000", 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8867e-05 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.827619+0000", 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.419e-05 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.827741+0000", 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121752 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.827985+0000", 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024428500000000001 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.828964+0000", 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00097892199999999991 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:456 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.841670+0000", 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.59492451800000001, 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020375950000000001, 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.505 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 456 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841670+0000", 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841670+0000", 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841673+0000", 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3159999999999999e-06 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841683+0000", 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0604e-05 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841712+0000", 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8416e-05 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841726+0000", 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4537e-05 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841840+0000", 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011335199999999999 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841977+0000", 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756599999999999 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.842352+0000", 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000374404 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.843312+0000", 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000960818 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.843632+0000", 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000319272 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.843674+0000", 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2373000000000002e-05 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.843708+0000", 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3936999999999997e-05 2024-05-06T00:26:51.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4364.0:473 3.13 e25/18 3:ccc1f26f:::rbd_header.110c2013632c:head v 25'6, mlcod=25'4)", 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.876072+0000", 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.56052285999999996, 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014253860000000001, 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.876072+0000", 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.876072+0000", 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.876074+0000", 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9829999999999999e-06 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.876088+0000", 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3987e-05 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.876122+0000", 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3875999999999999e-05 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.876140+0000", 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8423e-05 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.876289+0000", 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014871900000000001 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.876332+0000", 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2386999999999999e-05 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.877415+0000", 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001083318 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.877498+0000", 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2693000000000003e-05 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:1013 3.e 3:74163812:::rbd_data.110c17d674e9.0000000000000002:head [stat,delete] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.947162+0000", 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.489432855, 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032075910000000001, 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.515 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1013 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947162+0000", 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947162+0000", 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947165+0000", 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1499999999999999e-06 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947178+0000", 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2775999999999999e-05 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947210+0000", 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1486999999999999e-05 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947228+0000", 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947355+0000", 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126849 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947500+0000", 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144633 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947804+0000", 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030388700000000003 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.949610+0000", 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018066289999999999 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.950200+0000", 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00058935599999999997 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.950289+0000", 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8801999999999994e-05 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.950370+0000", 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1125999999999995e-05 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.520 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:1019 3.17 e25/18 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head v 25'33, mlcod=24'30)", 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.973953+0000", 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.46264160900000001, 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002045631, 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973953+0000", 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973953+0000", 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973954+0000", 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973962+0000", 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5739999999999998e-06 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973986+0000", 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4085999999999999e-05 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973999+0000", 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3353000000000001e-05 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.974119+0000", 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119826 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.974162+0000", 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2740000000000001e-05 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.975909+0000", 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747249 2024-05-06T00:26:51.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.975999+0000", 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0263e-05 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:1022 3.17 e25/18 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head v 25'35, mlcod=24'31)", 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.987176+0000", 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.44941947100000001, 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022568129999999999, 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.525 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987176+0000", 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987176+0000", 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987177+0000", 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.483e-06 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987189+0000", 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1841999999999999e-05 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987211+0000", 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2062e-05 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987222+0000", 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1184e-05 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987320+0000", 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8188000000000002e-05 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987358+0000", 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7623000000000002e-05 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.989337+0000", 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019787350000000001 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.989432+0000", 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5696e-05 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.529 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:558 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 139651801975840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.09451002, 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044008279999999999, 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 558 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342087+0000", 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8560000000000001e-06 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342095+0000", 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342119+0000", 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3632999999999999e-05 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342142+0000", 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3459999999999999e-05 2024-05-06T00:26:51.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342227+0000", 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4715999999999993e-05 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342358+0000", 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013112700000000001 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342590+0000", 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023153400000000001 2024-05-06T00:26:51.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.343979+0000", 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013895909999999999 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346305+0000", 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002325321 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346378+0000", 2024-05-06T00:26:51.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3276999999999997e-05 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346486+0000", 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010778299999999999 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4420.0:13 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 140209945667296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:51.425339+0000", 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.01125605, 2024-05-06T00:26:51.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024734610000000001, 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4420", 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 13 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425339+0000", 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425339+0000", 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425340+0000", 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0599999999999999e-07 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425345+0000", 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7679999999999999e-06 2024-05-06T00:26:51.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425359+0000", 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3954000000000001e-05 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425367+0000", 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1529999999999997e-06 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425434+0000", 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7712000000000005e-05 2024-05-06T00:26:51.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425557+0000", 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012269900000000001 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425791+0000", 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023337499999999999 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.427015+0000", 2024-05-06T00:26:51.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012240739999999999 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.427438+0000", 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042381999999999999 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.427604+0000", 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165489 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.427812+0000", 2024-05-06T00:26:51.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020851100000000001 2024-05-06T00:26:51.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.541 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:51.541 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:51.541 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:51.542 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-05-06T00:26:51.542 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:51.437+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:51.543 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:51.541+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:51.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:26:51.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:26:51.674 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:51.673+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:51.681 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:51.681 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:51.699 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:26:51.745 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:51.745+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:51.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:26:51.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:26:51.783 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:26:51.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:26:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:26:51.821 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:26:51.821 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:26:51.842 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:26:51.842 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:26:51.845 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:51.841+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:51.863 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:26:51.863 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:26:51.870 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:51.870 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:51.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:26:51.898 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:26:51.899 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:26:51.946 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:51.945+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:52.030 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:52.030 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:52.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:26:52.046 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:52.045+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:52.147 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:52.145+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:52.153 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0xd3e9c00 2024-05-06T00:26:52.158 INFO:teuthology.orchestra.run.smithi186.stdout:156 trunc from 0xc3c9400 to 0xd3e9c00 2024-05-06T00:26:52.162 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:52.162 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:52.177 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:26:52.191 INFO:teuthology.orchestra.run.smithi186.stdout:157 punch from 0xba0f800 to 0xba12000, (0x2800 bytes) 2024-05-06T00:26:52.245 INFO:teuthology.orchestra.run.smithi186.stdout:159 trunc from 0xd3e9c00 to 0x305f000 2024-05-06T00:26:52.248 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:52.249+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:52.314 INFO:teuthology.orchestra.run.smithi186.stdout:164 read 0x1b33e00 thru 0x1b39108 (0x5309 bytes) 2024-05-06T00:26:52.341 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:52.341 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:52.347 INFO:teuthology.orchestra.run.smithi186.stdout:165 write 0x3813000 thru 0x381f068 (0xc069 bytes) 2024-05-06T00:26:52.348 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:52.349+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:52.350 INFO:teuthology.orchestra.run.smithi186.stdout:166 read 0x1e53400 thru 0x1e5d140 (0x9d41 bytes) 2024-05-06T00:26:52.356 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:26:52.391 INFO:teuthology.orchestra.run.smithi186.stdout:169 write 0xa44ee00 thru 0xa451c68 (0x2e69 bytes) 2024-05-06T00:26:52.393 INFO:teuthology.orchestra.run.smithi186.stdout:170 punch from 0x4197c00 to 0x41a6e00, (0xf200 bytes) 2024-05-06T00:26:52.430 INFO:teuthology.orchestra.run.smithi186.stdout:171 write 0xd57a200 thru 0xd5872e6 (0xd0e7 bytes) 2024-05-06T00:26:52.433 INFO:teuthology.orchestra.run.smithi186.stdout:172 write 0x3275800 thru 0x327dbcd (0x83ce bytes) 2024-05-06T00:26:52.434 INFO:teuthology.orchestra.run.smithi186.stdout:173 write 0x48c0600 thru 0x48c234c (0x1d4d bytes) 2024-05-06T00:26:52.436 INFO:teuthology.orchestra.run.smithi186.stdout:176 write 0x7c8c600 thru 0x7c95f53 (0x9954 bytes) 2024-05-06T00:26:52.436 INFO:teuthology.orchestra.run.smithi186.stdout:178 read 0x14a2e00 thru 0x14a9a01 (0x6c02 bytes) 2024-05-06T00:26:52.437 INFO:teuthology.orchestra.run.smithi186.stdout:179 read 0x53a1c00 thru 0x53aa21a (0x861b bytes) 2024-05-06T00:26:52.450 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:52.449+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:52.478 INFO:teuthology.orchestra.run.smithi186.stdout:180 write 0xe1c1000 thru 0xe1c578f (0x4790 bytes) 2024-05-06T00:26:52.482 INFO:teuthology.orchestra.run.smithi186.stdout:183 write 0x92b9600 thru 0x92ba204 (0xc05 bytes) 2024-05-06T00:26:52.482 INFO:teuthology.orchestra.run.smithi186.stdout:185 punch from 0xaa04e00 to 0xaa0d800, (0x8a00 bytes) 2024-05-06T00:26:52.482 INFO:teuthology.orchestra.run.smithi186.stdout:186 punch from 0xdf7ec00 to 0xdf85000, (0x6400 bytes) 2024-05-06T00:26:52.483 INFO:teuthology.orchestra.run.smithi186.stdout:189 write 0x7ca3a00 thru 0x7ca3dad (0x3ae bytes) 2024-05-06T00:26:52.484 INFO:teuthology.orchestra.run.smithi186.stdout:193 read 0xc191800 thru 0xc1a0244 (0xea45 bytes) 2024-05-06T00:26:52.484 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:52.484 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:52.485 INFO:teuthology.orchestra.run.smithi186.stdout:195 read 0xb8d1e00 thru 0xb8d5e95 (0x4096 bytes) 2024-05-06T00:26:52.501 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:26:52.520 INFO:teuthology.orchestra.run.smithi186.stdout:197 trunc from 0xe1c5800 to 0x2a3cc00 2024-05-06T00:26:52.550 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:52.549+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:52.589 INFO:teuthology.orchestra.run.smithi186.stdout:198 trunc from 0x2a3cc00 to 0x32c8a00 2024-05-06T00:26:52.627 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:52.627 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:52.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:26:52.651 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:52.649+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:52.651 INFO:teuthology.orchestra.run.smithi186.stdout:199 write 0x4f95000 thru 0x4f9a48d (0x548e bytes) 2024-05-06T00:26:52.652 INFO:teuthology.orchestra.run.smithi186.stdout:200 read 0x21c6c00 thru 0x21d5a0d (0xee0e bytes) 2024-05-06T00:26:52.679 INFO:teuthology.orchestra.run.smithi186.stdout:203 trunc from 0x4f9a600 to 0x1494600 2024-05-06T00:26:52.727 INFO:teuthology.orchestra.run.smithi186.stdout:205 punch from 0xb19400 to 0xb20800, (0x7400 bytes) 2024-05-06T00:26:52.729 INFO:teuthology.orchestra.run.smithi186.stdout:207 read 0xc45c00 thru 0xc517df (0xbbe0 bytes) 2024-05-06T00:26:52.736 INFO:teuthology.orchestra.run.smithi186.stdout:209 read 0x11f600 thru 0x12304c (0x3a4d bytes) 2024-05-06T00:26:52.751 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:52.749+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:52.754 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:26:52.771 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:52.771 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:52.787 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:26:52.790 INFO:teuthology.orchestra.run.smithi186.stdout:210 write 0xbdb7c00 thru 0xbdc00e7 (0x84e8 bytes) 2024-05-06T00:26:52.795 INFO:teuthology.orchestra.run.smithi186.stdout:211 write 0xfdd400 thru 0xfe8126 (0xad27 bytes) 2024-05-06T00:26:52.797 INFO:teuthology.orchestra.run.smithi186.stdout:212 write 0x2f2600 thru 0x2fc8b8 (0xa2b9 bytes) 2024-05-06T00:26:52.802 INFO:teuthology.orchestra.run.smithi186.stdout:216 write 0x5bd0800 thru 0x5bd9fa1 (0x97a2 bytes) 2024-05-06T00:26:52.803 INFO:teuthology.orchestra.run.smithi186.stdout:219 read 0x3c7d400 thru 0x3c7f1b2 (0x1db3 bytes) 2024-05-06T00:26:52.803 INFO:teuthology.orchestra.run.smithi186.stdout:222 read 0x1d37a00 thru 0x1d406a0 (0x8ca1 bytes) 2024-05-06T00:26:52.805 INFO:teuthology.orchestra.run.smithi186.stdout:223 write 0x6d21200 thru 0x6d23840 (0x2641 bytes) 2024-05-06T00:26:52.805 INFO:teuthology.orchestra.run.smithi186.stdout:224 punch from 0x6120000 to 0x6128e00, (0x8e00 bytes) 2024-05-06T00:26:52.806 INFO:teuthology.orchestra.run.smithi186.stdout:225 punch from 0x4b41a00 to 0x4b4e000, (0xc600 bytes) 2024-05-06T00:26:52.806 INFO:teuthology.orchestra.run.smithi186.stdout:226 read 0x3f59a00 thru 0x3f5a08d (0x68e bytes) 2024-05-06T00:26:52.808 INFO:teuthology.orchestra.run.smithi186.stdout:230 trunc from 0xbdc0200 to 0xce57a00 2024-05-06T00:26:52.841 INFO:teuthology.orchestra.run.smithi186.stdout:231 punch from 0x235b800 to 0x236b600, (0xfe00 bytes) 2024-05-06T00:26:52.870 INFO:teuthology.orchestra.run.smithi186.stdout:232 write 0xd28b600 thru 0xd291b72 (0x6573 bytes) 2024-05-06T00:26:52.882 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:52.881+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:52.899 INFO:teuthology.orchestra.run.smithi186.stdout:233 write 0xd38f000 thru 0xd395328 (0x6329 bytes) 2024-05-06T00:26:52.902 INFO:teuthology.orchestra.run.smithi186.stdout:234 read 0x476da00 thru 0x477b2c8 (0xd8c9 bytes) 2024-05-06T00:26:52.923 INFO:teuthology.orchestra.run.smithi186.stdout:237 trunc from 0xd395400 to 0x570f000 2024-05-06T00:26:52.953 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:52.949+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:52.961 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:52.961 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:52.976 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:26:52.978 INFO:teuthology.orchestra.run.smithi186.stdout:240 write 0x49a1400 thru 0x49ae9b8 (0xd5b9 bytes) 2024-05-06T00:26:52.982 INFO:teuthology.orchestra.run.smithi186.stdout:241 read 0x381d000 thru 0x382a1e4 (0xd1e5 bytes) 2024-05-06T00:26:52.983 INFO:teuthology.orchestra.run.smithi186.stdout:244 read 0x45afe00 thru 0x45b3c4a (0x3e4b bytes) 2024-05-06T00:26:52.984 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0xd7f6000 2024-05-06T00:26:53.004 INFO:teuthology.orchestra.run.smithi186.stdout:246 trunc from 0x570f000 to 0xd7f6000 2024-05-06T00:26:53.024 INFO:teuthology.orchestra.run.smithi186.stdout:247 punch from 0x39400 to 0x40200, (0x6e00 bytes) 2024-05-06T00:26:53.055 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:53.053+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:53.055 INFO:teuthology.orchestra.run.smithi186.stdout:248 write 0xe158200 thru 0xe162291 (0xa092 bytes) 2024-05-06T00:26:53.091 INFO:teuthology.orchestra.run.smithi186.stdout:250 trunc from 0xe162400 to 0x1e05e00 2024-05-06T00:26:53.113 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:53.113 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:53.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:26:53.169 INFO:teuthology.orchestra.run.smithi186.stdout:252 trunc from 0x1e05e00 to 0x8387200 2024-05-06T00:26:53.172 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:53.173+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:53.210 INFO:teuthology.orchestra.run.smithi186.stdout:254 trunc from 0x8387200 to 0xd59200 2024-05-06T00:26:53.257 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:53.257+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:53.280 INFO:teuthology.orchestra.run.smithi186.stdout:256 write 0x5cef800 thru 0x5cfed12 (0xf513 bytes) 2024-05-06T00:26:53.280 INFO:teuthology.orchestra.run.smithi186.stdout:257 read 0x5344400 thru 0x534cccb (0x88cc bytes) 2024-05-06T00:26:53.281 INFO:teuthology.orchestra.run.smithi186.stdout:258 write 0x1e86600 thru 0x1e89afc (0x34fd bytes) 2024-05-06T00:26:53.281 INFO:teuthology.orchestra.run.smithi186.stdout:260 read 0x1289c00 thru 0x1299a4c (0xfe4d bytes) 2024-05-06T00:26:53.281 INFO:teuthology.orchestra.run.smithi186.stdout:261 write 0x7a0c00 thru 0x7aa368 (0x9769 bytes) 2024-05-06T00:26:53.288 INFO:teuthology.orchestra.run.smithi186.stdout:264 trunc from 0x5cfee00 to 0x2acb400 2024-05-06T00:26:53.306 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:53.307 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:53.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:26:53.348 INFO:teuthology.orchestra.run.smithi186.stdout:266 write 0x5369c00 thru 0x5369c2c (0x2d bytes) 2024-05-06T00:26:53.365 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:53.365+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:53.379 INFO:teuthology.orchestra.run.smithi186.stdout:268 write 0x88b0e00 thru 0x88badca (0x9fcb bytes) 2024-05-06T00:26:53.408 INFO:teuthology.orchestra.run.smithi186.stdout:269 write 0x9b35c00 thru 0x9b4462d (0xea2e bytes) 2024-05-06T00:26:53.413 INFO:teuthology.orchestra.run.smithi186.stdout:270 trunc from 0x9b44800 to 0xb02a000 2024-05-06T00:26:53.440 INFO:teuthology.orchestra.run.smithi186.stdout:273 read 0x368f800 thru 0x36964f9 (0x6cfa bytes) 2024-05-06T00:26:53.459 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:53.457+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:53.461 INFO:teuthology.orchestra.run.smithi186.stdout:276 trunc from 0xb02a000 to 0x37b4800 2024-05-06T00:26:53.501 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:53.502 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:53.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:26:53.532 INFO:teuthology.orchestra.run.smithi186.stdout:278 trunc from 0x37b4800 to 0x4c8000 2024-05-06T00:26:53.559 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0xe61a600 2024-05-06T00:26:53.559 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:53.557+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:53.598 INFO:teuthology.orchestra.run.smithi186.stdout:280 trunc from 0x4c8000 to 0xe61a600 2024-05-06T00:26:53.620 INFO:teuthology.orchestra.run.smithi186.stdout:281 punch from 0x943e600 to 0x944c800, (0xe200 bytes) 2024-05-06T00:26:53.622 INFO:teuthology.orchestra.run.smithi186.stdout:285 write 0x898c600 thru 0x89964ab (0x9eac bytes) 2024-05-06T00:26:53.649 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:53.649 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:53.654 INFO:teuthology.orchestra.run.smithi186.stdout:290 trunc from 0xe61a600 to 0x554c00 2024-05-06T00:26:53.660 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:53.657+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:53.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:26:53.701 INFO:teuthology.orchestra.run.smithi186.stdout:291 punch from 0x411000 to 0x41ce00, (0xbe00 bytes) 2024-05-06T00:26:53.756 INFO:teuthology.orchestra.run.smithi186.stdout:293 write 0xb0f1e00 thru 0xb0fa637 (0x8838 bytes) 2024-05-06T00:26:53.756 INFO:teuthology.orchestra.run.smithi186.stdout:295 read 0x26c8800 thru 0x26d61d9 (0xd9da bytes) 2024-05-06T00:26:53.756 INFO:teuthology.orchestra.run.smithi186.stdout:297 punch from 0xa075400 to 0xa083600, (0xe200 bytes) 2024-05-06T00:26:53.757 INFO:teuthology.orchestra.run.smithi186.stdout:298 read 0x3cc3600 thru 0x3cc7e7a (0x487b bytes) 2024-05-06T00:26:53.757 INFO:teuthology.orchestra.run.smithi186.stdout:299 read 0x491fe00 thru 0x4920cae (0xeaf bytes) 2024-05-06T00:26:53.757 INFO:teuthology.orchestra.run.smithi186.stdout:300 read 0x521d200 thru 0x5223d60 (0x6b61 bytes) 2024-05-06T00:26:53.757 INFO:teuthology.orchestra.run.smithi186.stdout:303 read 0x3286e00 thru 0x3290f67 (0xa168 bytes) 2024-05-06T00:26:53.757 INFO:teuthology.orchestra.run.smithi186.stdout:305 write 0x663000 thru 0x66cd61 (0x9d62 bytes) 2024-05-06T00:26:53.761 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:53.761+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:53.780 INFO:teuthology.orchestra.run.smithi186.stdout:306 write 0xd801e00 thru 0xd803dfc (0x1ffd bytes) 2024-05-06T00:26:53.782 INFO:teuthology.orchestra.run.smithi186.stdout:308 read 0xd79f800 thru 0xd7a8360 (0x8b61 bytes) 2024-05-06T00:26:53.796 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:53.796 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:53.800 INFO:teuthology.orchestra.run.smithi186.stdout:310 trunc from 0xd803e00 to 0x40f2a00 2024-05-06T00:26:53.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:26:53.851 INFO:teuthology.orchestra.run.smithi186.stdout:311 punch from 0x1d1c200 to 0x1d29a00, (0xd800 bytes) 2024-05-06T00:26:53.854 INFO:teuthology.orchestra.run.smithi186.stdout:314 punch from 0x35a4800 to 0x35a9000, (0x4800 bytes) 2024-05-06T00:26:53.854 INFO:teuthology.orchestra.run.smithi186.stdout:316 punch from 0x35cda00 to 0x35d5c00, (0x8200 bytes) 2024-05-06T00:26:53.854 INFO:teuthology.orchestra.run.smithi186.stdout:317 read 0x338ae00 thru 0x3396fd2 (0xc1d3 bytes) 2024-05-06T00:26:53.887 INFO:teuthology.orchestra.run.smithi186.stdout:319 write 0x94fc000 thru 0x95051aa (0x91ab bytes) 2024-05-06T00:26:53.889 INFO:teuthology.orchestra.run.smithi186.stdout:320 read 0x5913000 thru 0x5919f45 (0x6f46 bytes) 2024-05-06T00:26:53.900 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:53.901+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:53.910 INFO:teuthology.orchestra.run.smithi186.stdout:323 trunc from 0x9505200 to 0x7fc000 2024-05-06T00:26:53.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:26:53.963 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:53.961+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:53.973 INFO:teuthology.orchestra.run.smithi186.stdout:324 punch from 0x212c00 to 0x216a00, (0x3e00 bytes) 2024-05-06T00:26:53.974 INFO:teuthology.orchestra.run.smithi186.stdout:325 clone 3 order 22 su 131072 sc 7 2024-05-06T00:26:53.988 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:53.988 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:54.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:26:54.074 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:26:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:26:54.088 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:54.089+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:54.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:26:54.091 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:26:54.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:26:54.108 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:26:54.126 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:26:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:26:54.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:26:54.145 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:26:54.165 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:54.165+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:54.165 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:26:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:26:54.170 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:26:54.170 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:54.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:26:54.266 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:54.265+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:54.352 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:26:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:26:54.353 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:54.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:26:54.367 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:54.365+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:54.467 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:54.465+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:54.531 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:26:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:26:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:26:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:10 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.8048603619999994, 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074329959999999999, 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 10 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:26:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725266+0000", 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8830000000000002e-06 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725276+0000", 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.042e-05 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725297+0000", 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0927000000000001e-05 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725314+0000", 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7266e-05 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725442+0000", 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012823899999999999 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725452+0000", 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.944e-06 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729257+0000", 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038050530000000001 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729483+0000", 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022601199999999999 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729822+0000", 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033856400000000002 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.731316+0000", 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014941209999999999 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732407+0000", 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010905820000000001 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732553+0000", 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014660899999999999 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732696+0000", 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014237599999999999 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:12 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.737020+0000", 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7931032689999995, 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003051893, 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 12 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737020+0000", 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737020+0000", 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737022+0000", 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5459999999999998e-06 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737036+0000", 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3886999999999999e-05 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737065+0000", 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9105999999999999e-05 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737081+0000", 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5307000000000001e-05 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737219+0000", 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013803200000000001 2024-05-06T00:26:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737454+0000", 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023531499999999999 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.737835+0000", 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038118700000000001 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.739250+0000", 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014146919999999999 2024-05-06T00:26:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.739911+0000", 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066157200000000003 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.740002+0000", 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0172000000000003e-05 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.740072+0000", 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0077e-05 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:25 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 139651801167792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.765469+0000", 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7646545190000005, 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026801590000000001, 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 25 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765469+0000", 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765469+0000", 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765471+0000", 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2730000000000001e-06 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765481+0000", 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0190000000000001e-05 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765512+0000", 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1013999999999998e-05 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765530+0000", 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8318999999999999e-05 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765689+0000", 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015869500000000001 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.765825+0000", 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136456 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.766057+0000", 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023154499999999999 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.767205+0000", 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011481149999999999 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.767887+0000", 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068138699999999999 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.767979+0000", 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2289999999999999e-05 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.768149+0000", 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000169875 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4467.0:12 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 139796487965376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.857907+0000", 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6722156649999995, 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029019380000000002, 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4467", 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3050489859", 2024-05-06T00:26:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 12 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857907+0000", 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857907+0000", 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857909+0000", 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6330000000000001e-06 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857918+0000", 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4399999999999994e-06 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857941+0000", 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2903000000000001e-05 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.857954+0000", 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2320000000000001e-05 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.858069+0000", 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011578600000000001 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.858296+0000", 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022668800000000001 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.858770+0000", 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047380399999999998 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.860231+0000", 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014605849999999999 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.860629+0000", 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039847999999999999 2024-05-06T00:26:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.860710+0000", 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0646e-05 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.860809+0000", 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9653000000000003e-05 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:556 3.17 e23/18 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head v 23'22, mlcod=23'20)", 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.8429584019999998, 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039476930000000004, 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687167+0000", 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9269999999999999e-06 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687457+0000", 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029068099999999999 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687495+0000", 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7870000000000002e-05 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687514+0000", 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8419000000000001e-05 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687650+0000", 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136966 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687692+0000", 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1029999999999998e-05 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690995+0000", 2024-05-06T00:26:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033034140000000002 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691112+0000", 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117386 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:557 3.17 e23/18 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head v 23'23, mlcod=23'20)", 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.842557115, 2024-05-06T00:26:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039228129999999998, 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687567+0000", 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2869999999999999e-06 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687619+0000", 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2119e-05 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687640+0000", 2024-05-06T00:26:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1123000000000001e-05 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687656+0000", 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.523e-05 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688969+0000", 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00131369 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.689027+0000", 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000001e-05 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691373+0000", 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023457819999999998 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691489+0000", 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116042 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:558 3.17 e23/18 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head v 23'24, mlcod=23'20)", 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.8419000800000003, 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037091110000000002, 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688225+0000", 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4940000000000002e-06 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688325+0000", 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9679000000000006e-05 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688356+0000", 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1313000000000003e-05 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688377+0000", 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0970000000000001e-05 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690069+0000", 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016917360000000001 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690106+0000", 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6912999999999997e-05 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691873+0000", 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017671200000000001 2024-05-06T00:26:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691932+0000", 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8885999999999998e-05 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:690 3.2 e24/18 3:405a3ab9:::rbd_data.110c17d674e9.000000000000000a:head v 24'3, mlcod=23'1)", 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.004869+0000", 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5252535360000001, 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002965526, 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.004869+0000", 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.004869+0000", 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.004870+0000", 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7700000000000001e-07 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005118+0000", 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024812499999999997 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005142+0000", 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3697999999999999e-05 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005154+0000", 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.218e-05 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005238+0000", 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4537999999999996e-05 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.005272+0000", 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3408000000000003e-05 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.007786+0000", 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025146259999999998 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.007835+0000", 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:694 3.2 e24/18 3:405a3ab9:::rbd_data.110c17d674e9.000000000000000a:head v 24'5, mlcod=24'3)", 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.031413+0000", 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4987097900000004, 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015943260000000001, 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031413+0000", 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031413+0000", 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031415+0000", 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6449999999999999e-06 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031426+0000", 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1260000000000001e-05 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031446+0000", 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9942999999999999e-05 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031456+0000", 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8209999999999997e-06 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031584+0000", 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012832100000000001 2024-05-06T00:26:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.031644+0000", 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0123e-05 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.032933+0000", 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012884859999999999 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.033008+0000", 2024-05-06T00:26:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4727000000000005e-05 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:707 3.e 3:74163812:::rbd_data.110c17d674e9.0000000000000002:head [stat,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.040217+0000", 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.489906103, 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002539829, 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 707 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040217+0000", 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040217+0000", 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040220+0000", 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5370000000000002e-06 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040234+0000", 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3439e-05 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040269+0000", 2024-05-06T00:26:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5240000000000001e-05 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040285+0000", 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6317e-05 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040416+0000", 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130359 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040569+0000", 2024-05-06T00:26:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015351199999999999 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.040821+0000", 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025124199999999999 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.041812+0000", 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00099124499999999997 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.042686+0000", 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087427800000000003 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.042723+0000", 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7237000000000003e-05 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.042757+0000", 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3423000000000002e-05 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:426 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.781779+0000", 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.7483436829999999, 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029378160000000002, 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 426 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781779+0000", 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781779+0000", 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781783+0000", 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2669999999999998e-06 2024-05-06T00:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781798+0000", 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5173e-05 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781832+0000", 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4341999999999997e-05 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781849+0000", 2024-05-06T00:26:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.658e-05 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781948+0000", 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9444000000000005e-05 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.781959+0000", 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0427e-05 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.784464+0000", 2024-05-06T00:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025050789999999999 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.784562+0000", 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8096000000000004e-05 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.784717+0000", 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155408 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:456 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.841670+0000", 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.688452549, 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020375950000000001, 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 456 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841670+0000", 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841670+0000", 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841673+0000", 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3159999999999999e-06 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841683+0000", 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0604e-05 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841712+0000", 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8416e-05 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841726+0000", 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4537e-05 2024-05-06T00:26:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841840+0000", 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011335199999999999 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.841977+0000", 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756599999999999 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.842352+0000", 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000374404 2024-05-06T00:26:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.843312+0000", 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000960818 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.843632+0000", 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000319272 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.843674+0000", 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2373000000000002e-05 2024-05-06T00:26:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.843708+0000", 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3936999999999997e-05 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:1013 3.e 3:74163812:::rbd_data.110c17d674e9.0000000000000002:head [stat,delete] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.947162+0000", 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.582960886, 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032075910000000001, 2024-05-06T00:26:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1013 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947162+0000", 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947162+0000", 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947165+0000", 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1499999999999999e-06 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947178+0000", 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2775999999999999e-05 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947210+0000", 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1486999999999999e-05 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947228+0000", 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947355+0000", 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126849 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947500+0000", 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144633 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.947804+0000", 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030388700000000003 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.949610+0000", 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018066289999999999 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.950200+0000", 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00058935599999999997 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.950289+0000", 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8801999999999994e-05 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.950370+0000", 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1125999999999995e-05 2024-05-06T00:26:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:1019 3.17 e25/18 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head v 25'33, mlcod=24'30)", 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.973953+0000", 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5561696400000002, 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002045631, 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973953+0000", 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973953+0000", 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973954+0000", 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973962+0000", 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5739999999999998e-06 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973986+0000", 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4085999999999999e-05 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.973999+0000", 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3353000000000001e-05 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.974119+0000", 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119826 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.974162+0000", 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2740000000000001e-05 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.975909+0000", 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747249 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.975999+0000", 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0263e-05 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:1022 3.17 e25/18 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head v 25'35, mlcod=24'31)", 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.987176+0000", 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5429475020000001, 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022568129999999999, 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987176+0000", 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987176+0000", 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987177+0000", 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.483e-06 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987189+0000", 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1841999999999999e-05 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987211+0000", 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2062e-05 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987222+0000", 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1184e-05 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987320+0000", 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8188000000000002e-05 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.987358+0000", 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7623000000000002e-05 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.989337+0000", 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019787350000000001 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.989432+0000", 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5696e-05 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:558 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 139651801975840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.1880380509999999, 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044008279999999999, 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 558 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342087+0000", 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8560000000000001e-06 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342095+0000", 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342119+0000", 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3632999999999999e-05 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342142+0000", 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3459999999999999e-05 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342227+0000", 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4715999999999993e-05 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342358+0000", 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013112700000000001 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342590+0000", 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023153400000000001 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.343979+0000", 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013895909999999999 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346305+0000", 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002325321 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346378+0000", 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3276999999999997e-05 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346486+0000", 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010778299999999999 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4420.0:13 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 140209945667296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:51.425339+0000", 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.104784081, 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024734610000000001, 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4420", 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 13 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425339+0000", 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425339+0000", 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425340+0000", 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0599999999999999e-07 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425345+0000", 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7679999999999999e-06 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425359+0000", 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3954000000000001e-05 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425367+0000", 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1529999999999997e-06 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425434+0000", 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7712000000000005e-05 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425557+0000", 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012269900000000001 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.425791+0000", 2024-05-06T00:26:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023337499999999999 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.427015+0000", 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012240739999999999 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.427438+0000", 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042381999999999999 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.427604+0000", 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165489 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.427812+0000", 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020851100000000001 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4420.0:58 3.e 3:7370c184:::rbd_data.110cdb7de3c5.000000000000001d:head [read 1007616~40960] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:51.497632+0000", 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.032490862, 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019924320000000001, 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4420", 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 58 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.497632+0000", 2024-05-06T00:26:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.497632+0000", 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.497633+0000", 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7300000000000001e-07 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.497634+0000", 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6789999999999999e-06 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.497639+0000", 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1449999999999999e-06 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.497644+0000", 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3019999999999996e-06 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.499139+0000", 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001495686 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.499351+0000", 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021189200000000001 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.499625+0000", 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027325500000000003 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4420.0:61 3.e 3:7370c184:::rbd_data.110cdb7de3c5.000000000000001d:head [read 1126400~40960] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:51.500283+0000", 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.0298403920000001, 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016253540000000001, 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4420", 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 61 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500283+0000", 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500283+0000", 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500283+0000", 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4600000000000004e-07 2024-05-06T00:26:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500287+0000", 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7639999999999999e-06 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500302+0000", 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4696e-05 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500310+0000", 2024-05-06T00:26:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9999999999999996e-06 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.501449+0000", 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001138661 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.501646+0000", 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019772299999999999 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.501908+0000", 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000261764 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4420.0:62 3.e 3:7370c184:::rbd_data.110cdb7de3c5.000000000000001d:head [read 1171456~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:51.500322+0000", 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.0298013959999999, 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001760225, 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:26:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4420", 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 62 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500322+0000", 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500322+0000", 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500322+0000", 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500326+0000", 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2100000000000002e-06 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500334+0000", 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9299999999999992e-06 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.500342+0000", 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1029999999999996e-06 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.501940+0000", 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015981910000000001 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.501996+0000", 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301000000000002e-05 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:26:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.502082+0000", 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5805999999999998e-05 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:26:54.633 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:54.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:26:54.635 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:54.633+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:54.736 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:54.733+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:54.772 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:54.772 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:54.772 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:54.772 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:54.776 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:26:54.788 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:26:54.837 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:54.837+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:54.938 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:54.937+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:54.942 INFO:teuthology.orchestra.run.smithi186.stdout:flattening image image_client.0-clone2 2024-05-06T00:26:54.951 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:54.951 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:54.951 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:54.952 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:54.952 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:54.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:26:55.038 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:55.037+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:55.087 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:23 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.279833+0000", 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.806600381999999, 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044081570000000002, 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.088 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279833+0000", 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279833+0000", 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279834+0000", 2024-05-06T00:26:55.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.874e-06 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279949+0000", 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011439599999999999 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279978+0000", 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9386e-05 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.279992+0000", 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3613e-05 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.280135+0000", 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014314899999999999 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.280193+0000", 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8122999999999999e-05 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.284143+0000", 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039495930000000004 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.284241+0000", 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8023000000000007e-05 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4174.0:50 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'27)", 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:30.395131+0000", 2024-05-06T00:26:55.092 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.691302214, 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054106950000000001, 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395131+0000", 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395131+0000", 2024-05-06T00:26:55.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395132+0000", 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1519999999999999e-06 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395203+0000", 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1229000000000005e-05 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395220+0000", 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6741000000000001e-05 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395249+0000", 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9300000000000001e-05 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395310+0000", 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0782000000000002e-05 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.395341+0000", 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1424000000000003e-05 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.400493+0000", 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051515490000000001 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:30.400541+0000", 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8517999999999999e-05 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:545 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.679150+0000", 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4072832100000001, 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052292340000000001, 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 545 2024-05-06T00:26:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679150+0000", 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679150+0000", 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679150+0000", 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4700000000000002e-07 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679154+0000", 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9960000000000004e-06 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679162+0000", 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3370000000000002e-06 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679165+0000", 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.484e-06 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679195+0000", 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0246000000000001e-05 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679241+0000", 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5849000000000001e-05 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679455+0000", 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021342299999999999 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.680931+0000", 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001476932 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684330+0000", 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033990470000000001 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684358+0000", 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7727000000000001e-05 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684379+0000", 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0746000000000001e-05 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:546 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.679245+0000", 2024-05-06T00:26:55.102 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.407187779, 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052773170000000001, 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 546 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679245+0000", 2024-05-06T00:26:55.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679245+0000", 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679246+0000", 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4199999999999995e-07 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679250+0000", 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679256+0000", 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679262+0000", 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3899999999999998e-06 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679779+0000", 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051624600000000004 2024-05-06T00:26:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679835+0000", 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6684999999999997e-05 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.680028+0000", 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019233100000000001 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.680938+0000", 2024-05-06T00:26:55.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091022100000000001 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684429+0000", 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034911539999999998 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684484+0000", 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5392000000000001e-05 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684522+0000", 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8120000000000001e-05 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:547 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.679448+0000", 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4069851330000001, 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051655360000000001, 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 547 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679448+0000", 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679448+0000", 2024-05-06T00:26:55.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679448+0000", 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679451+0000", 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679459+0000", 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.028e-06 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.679462+0000", 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2490000000000001e-06 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.680310+0000", 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00084798100000000004 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.680401+0000", 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0815000000000004e-05 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.680688+0000", 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028719599999999998 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.681222+0000", 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00053406199999999997 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684557+0000", 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033354909999999999 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684589+0000", 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1242999999999999e-05 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684613+0000", 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4712e-05 2024-05-06T00:26:55.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:551 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.681281+0000", 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4051520530000001, 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043174830000000004, 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 551 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.681281+0000", 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.681281+0000", 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.681281+0000", 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1600000000000002e-07 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.681283+0000", 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0660000000000002e-06 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.681288+0000", 2024-05-06T00:26:55.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5240000000000002e-06 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.681291+0000", 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1549999999999999e-06 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.682148+0000", 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00085699699999999997 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.682194+0000", 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6442e-05 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.682408+0000", 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021342 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685282+0000", 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028736539999999998 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685464+0000", 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000182838 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685556+0000", 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1556000000000002e-05 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685598+0000", 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2415e-05 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:556 3.17 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write 15585792~56320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.684975+0000", 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4014574599999996, 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004701439, 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 556 2024-05-06T00:26:55.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684975+0000", 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684975+0000", 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.684976+0000", 2024-05-06T00:26:55.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.68e-07 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685018+0000", 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2064999999999998e-05 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685025+0000", 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.844e-06 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685031+0000", 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7749999999999998e-06 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685084+0000", 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3267999999999999e-05 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685171+0000", 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7046999999999995e-05 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-06T00:26:55.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685267+0000", 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5922999999999996e-05 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686736+0000", 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001468918 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.689623+0000", 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028869439999999998 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.689652+0000", 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8815000000000001e-05 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.689677+0000", 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5272000000000001e-05 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:557 3.17 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write 15757824~18944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.685075+0000", 2024-05-06T00:26:55.127 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4013575700000001, 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049041609999999998, 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 557 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685075+0000", 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685075+0000", 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685076+0000", 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6700000000000003e-07 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685091+0000", 2024-05-06T00:26:55.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5235000000000001e-05 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685099+0000", 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8509999999999999e-06 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685105+0000", 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.694e-06 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685675+0000", 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00057003900000000005 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685719+0000", 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4239000000000001e-05 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685799+0000", 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9696000000000004e-05 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.688893+0000", 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003094477 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.689932+0000", 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010384350000000001 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.689958+0000", 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6707999999999999e-05 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.689980+0000", 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.122e-05 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:558 3.17 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write 5419008~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4013188330000004, 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065722799999999998, 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 558 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685115+0000", 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9399999999999995e-07 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685137+0000", 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2628000000000001e-05 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685145+0000", 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6310000000000003e-06 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685152+0000", 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3889999999999999e-06 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686121+0000", 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096883500000000003 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686199+0000", 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7911000000000007e-05 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686364+0000", 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000164754 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.690386+0000", 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040223309999999996 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691639+0000", 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012525640000000001 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691667+0000", 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8084000000000001e-05 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691686+0000", 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9658999999999999e-05 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.138 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:686 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.996424+0000", 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.0900092050000003, 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042852430000000002, 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 686 2024-05-06T00:26:55.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996424+0000", 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996424+0000", 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996426+0000", 2024-05-06T00:26:55.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.616e-06 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996439+0000", 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2227e-05 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996469+0000", 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0533000000000001e-05 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996484+0000", 2024-05-06T00:26:55.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4837e-05 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996601+0000", 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117392 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996764+0000", 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000162215 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.997326+0000", 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00056196600000000001 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.000532+0000", 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032069350000000002 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.000559+0000", 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6047e-05 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.000643+0000", 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4962000000000004e-05 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.000709+0000", 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5512999999999994e-05 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:687 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:55.144 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.996517+0000", 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.0899163229999997, 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049824079999999998, 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 687 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996517+0000", 2024-05-06T00:26:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996517+0000", 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996517+0000", 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6300000000000004e-07 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996522+0000", 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9099999999999996e-06 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996534+0000", 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1197e-05 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.996542+0000", 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0129999999999993e-06 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.998340+0000", 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017979160000000001 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.998601+0000", 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000261792 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.999732+0000", 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001130374 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001221+0000", 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001488886 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001263+0000", 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2069e-05 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001393+0000", 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130722 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001499+0000", 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105666 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:689 3.6 3:64805dc9:::rbd_data.110c17d674e9.000000000000000e:head [set-alloc-hint object_size 16777216 write_size 16777216,write 3723264~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.0852333820000002, 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611419999999999, 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 689 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:26:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001201+0000", 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3859999999999999e-06 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001252+0000", 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1165999999999998e-05 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001270+0000", 2024-05-06T00:26:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7813e-05 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001281+0000", 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1209999999999999e-05 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001382+0000", 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010090300000000001 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001599+0000", 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021734900000000001 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001815+0000", 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021530799999999999 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.003840+0000", 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020251010000000001 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007312+0000", 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034724130000000001 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007390+0000", 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7810000000000002e-05 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007461+0000", 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0683000000000001e-05 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:695 3.12 3:4944ac60:::rbd_data.110c17d674e9.000000000000000b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.028789+0000", 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.0576439989999997, 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050745410000000001, 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 695 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.156 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028789+0000", 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028789+0000", 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028792+0000", 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9170000000000002e-06 2024-05-06T00:26:55.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028801+0000", 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1970000000000002e-06 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028825+0000", 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3563000000000001e-05 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028838+0000", 2024-05-06T00:26:55.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2962999999999999e-05 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028975+0000", 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137229 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029186+0000", 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021123499999999999 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029476+0000", 2024-05-06T00:26:55.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029021100000000002 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.031645+0000", 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021684 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.033614+0000", 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019693779999999999 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.033759+0000", 2024-05-06T00:26:55.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014503300000000001 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.033863+0000", 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104415 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:697 3.6 3:64805dc9:::rbd_data.110c17d674e9.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.028894+0000", 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.0575392399999997, 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057253670000000003, 2024-05-06T00:26:55.161 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 697 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028894+0000", 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028894+0000", 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028896+0000", 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8700000000000001e-06 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028903+0000", 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0570000000000004e-06 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028924+0000", 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1112999999999999e-05 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.028940+0000", 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6257e-05 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029041+0000", 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100852 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029217+0000", 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017575199999999999 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029497+0000", 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028074099999999999 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.031037+0000", 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015401080000000001 2024-05-06T00:26:55.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034434+0000", 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033968219999999999 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034513+0000", 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039e-05 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034619+0000", 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105756 2024-05-06T00:26:55.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:852 3.1b 3:da0a6ebf:::rbd_data.110c17d674e9.0000000000000003:head [write 5349376~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.153986+0000", 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9324466019999997, 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051664559999999998, 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.167 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 852 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.153986+0000", 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.153986+0000", 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.153988+0000", 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.604e-06 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.154011+0000", 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3388999999999999e-05 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.154030+0000", 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8603000000000001e-05 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.154040+0000", 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8099999999999992e-06 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.154114+0000", 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4302999999999997e-05 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.154231+0000", 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117346 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.154457+0000", 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022534200000000001 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.155960+0000", 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001502911 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.159058+0000", 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030987100000000002 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.159112+0000", 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3281e-05 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.159153+0000", 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1156999999999999e-05 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:1007 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.919388+0000", 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1670446749999996, 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041976569999999996, 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1007 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.919388+0000", 2024-05-06T00:26:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.919388+0000", 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.919391+0000", 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3630000000000001e-06 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.919402+0000", 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1537e-05 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.919432+0000", 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9923000000000001e-05 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.919447+0000", 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4517e-05 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.919593+0000", 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000146229 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.919848+0000", 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254868 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.920933+0000", 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001085761 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.922732+0000", 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001798329 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.923307+0000", 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00057524300000000002 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.923468+0000", 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016074199999999999 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.923586+0000", 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118145 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:1017 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.962425+0000", 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1240078640000002, 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044600389999999998, 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1017 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.962425+0000", 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.962425+0000", 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.962428+0000", 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5969999999999999e-06 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.962440+0000", 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1827e-05 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.962470+0000", 2024-05-06T00:26:55.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0766000000000003e-05 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.962485+0000", 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5006e-05 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.962619+0000", 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133239 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.962881+0000", 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026222499999999999 2024-05-06T00:26:55.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.963965+0000", 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001083865 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.966230+0000", 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022653090000000001 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.966579+0000", 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034866399999999999 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.966753+0000", 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000174358 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.966885+0000", 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013218299999999999 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:1020 3.1b 3:df802a3a:::rbd_object_map.110c17d674e9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.975314+0000", 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1111189469999996, 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044348429999999999, 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1020 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.184 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.975314+0000", 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.975314+0000", 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.975316+0000", 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3800000000000001e-06 2024-05-06T00:26:55.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.975328+0000", 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.172e-05 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.975358+0000", 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0300000000000001e-05 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.975375+0000", 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6116999999999998e-05 2024-05-06T00:26:55.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.975510+0000", 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013596499999999999 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.975760+0000", 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024990199999999998 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.976851+0000", 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001090518 2024-05-06T00:26:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.978650+0000", 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017995960000000001 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.979470+0000", 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00081963999999999995 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.979628+0000", 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015818599999999999 2024-05-06T00:26:55.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.979749+0000", 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120519 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4408.0:2223 3.6 3:65d72c0c:::rbd_object_map.110c2013632c:head [read 0~35] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:54.772796+0000", 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.31363689500000003, 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044412230000000002, 2024-05-06T00:26:55.189 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4408", 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1905298218", 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2223 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.772796+0000", 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.772796+0000", 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.772799+0000", 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7499999999999999e-06 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.772811+0000", 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2367e-05 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.772843+0000", 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1483000000000003e-05 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.772859+0000", 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6569999999999999e-05 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.773039+0000", 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000179815 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.773052+0000", 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2937e-05 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.776917+0000", 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003864896 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.777055+0000", 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013796899999999999 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.777237+0000", 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018243600000000001 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4408.0:2225 3.6 3:65d72c0c:::rbd_object_map.110c2013632c:head [call lock.assert_locked in=17b,call] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:54.781548+0000", 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.30488520499999999, 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053061949999999997, 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.194 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4408", 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1905298218", 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2225 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.781548+0000", 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.781548+0000", 2024-05-06T00:26:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.781551+0000", 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.517e-06 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.781565+0000", 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3976e-05 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.781599+0000", 2024-05-06T00:26:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4116000000000003e-05 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.781616+0000", 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6969999999999998e-05 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.781741+0000", 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000124809 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.781907+0000", 2024-05-06T00:26:55.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016573900000000001 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782335+0000", 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042791999999999998 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.786239+0000", 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039039029999999998 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.786661+0000", 2024-05-06T00:26:55.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042214999999999999 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.786758+0000", 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7329000000000003e-05 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.786854+0000", 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5766000000000004e-05 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.199 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.200 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:55.201 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:26:55.201 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:55.201+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:55.301 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:55.301+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:55.396 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:26:55.403 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-05-06T00:26:55.403 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:55.401+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:55.504 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:55.505+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:55.563 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:55.563 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:55.563 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:55.563 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:55.563 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:55.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:26:55.605 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:55.605+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:55.633 INFO:teuthology.orchestra.run.smithi186.stdout:328 write 0xdc70800 thru 0xdc73959 (0x315a bytes) 2024-05-06T00:26:55.647 INFO:teuthology.orchestra.run.smithi186.stdout:330 trunc from 0xdc73a00 to 0x7932a00 2024-05-06T00:26:55.680 INFO:teuthology.orchestra.run.smithi186.stdout:332 write 0x4ec7800 thru 0x4ed22b0 (0xaab1 bytes) 2024-05-06T00:26:55.681 INFO:teuthology.orchestra.run.smithi186.stdout:333 read 0x2351800 thru 0x2356e32 (0x5633 bytes) 2024-05-06T00:26:55.681 INFO:teuthology.orchestra.run.smithi186.stdout:336 punch from 0x5d9d800 to 0x5da8200, (0xaa00 bytes) 2024-05-06T00:26:55.682 INFO:teuthology.orchestra.run.smithi186.stdout:337 read 0x3d28200 thru 0x3d359f3 (0xd7f4 bytes) 2024-05-06T00:26:55.682 INFO:teuthology.orchestra.run.smithi186.stdout:338 read 0x5278e00 thru 0x527eb25 (0x5d26 bytes) 2024-05-06T00:26:55.682 INFO:teuthology.orchestra.run.smithi186.stdout:339 punch from 0x5eb0a00 to 0x5eb5000, (0x4600 bytes) 2024-05-06T00:26:55.683 INFO:teuthology.orchestra.run.smithi186.stdout:340 read 0x7077c00 thru 0x707ea62 (0x6e63 bytes) 2024-05-06T00:26:55.683 INFO:teuthology.orchestra.run.smithi186.stdout:343 write 0x2f7e200 thru 0x2f88243 (0xa044 bytes) 2024-05-06T00:26:55.683 INFO:teuthology.orchestra.run.smithi186.stdout:344 read 0x45bf000 thru 0x45ca2e8 (0xb2e9 bytes) 2024-05-06T00:26:55.706 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:55.705+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:55.716 INFO:teuthology.orchestra.run.smithi186.stdout:345 write 0xa528800 thru 0xa53650e (0xdd0f bytes) 2024-05-06T00:26:55.716 INFO:teuthology.orchestra.run.smithi186.stdout:347 write 0x245c400 thru 0x245f6e2 (0x32e3 bytes) 2024-05-06T00:26:55.716 INFO:teuthology.orchestra.run.smithi186.stdout:349 read 0x75aec00 thru 0x75b146a (0x286b bytes) 2024-05-06T00:26:55.716 INFO:teuthology.orchestra.run.smithi186.stdout:350 punch from 0x9b71a00 to 0x9b75800, (0x3e00 bytes) 2024-05-06T00:26:55.717 INFO:teuthology.orchestra.run.smithi186.stdout:351 punch from 0x3ddd000 to 0x3dec000, (0xf000 bytes) 2024-05-06T00:26:55.717 INFO:teuthology.orchestra.run.smithi186.stdout:352 write 0x4791600 thru 0x479f96a (0xe36b bytes) 2024-05-06T00:26:55.720 INFO:teuthology.orchestra.run.smithi186.stdout:353 trunc from 0xa536600 to 0xbbcf400 2024-05-06T00:26:55.721 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:55.721 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:26:55.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:26:55.721 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:26:55.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.721 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:690 3.2 3:405a3ab9:::rbd_data.110c17d674e9.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write 9261056~53248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:55.721 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.001857+0000", 2024-05-06T00:26:55.721 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.7149658140000001, 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048177300000000001, 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 690 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001857+0000", 2024-05-06T00:26:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001857+0000", 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001860+0000", 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0000000000000001e-06 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001982+0000", 2024-05-06T00:26:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121999 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002019+0000", 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7533000000000001e-05 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002044+0000", 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4709e-05 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002174+0000", 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013002599999999999 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002545+0000", 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037078099999999999 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002988+0000", 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044360599999999998 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.004395+0000", 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014062790000000001 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.006475+0000", 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020805849999999998 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.006573+0000", 2024-05-06T00:26:55.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8302999999999997e-05 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.006674+0000", 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010090900000000001 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4412.0:689 3.6 e24/18 3:64805dc9:::rbd_data.110c17d674e9.000000000000000e:head v 24'1, mlcod=0'0)", 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.002177+0000", 2024-05-06T00:26:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.7146454899999997, 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048587960000000003, 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002177+0000", 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002177+0000", 2024-05-06T00:26:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002178+0000", 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8599999999999997e-07 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002215+0000", 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6813999999999997e-05 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002246+0000", 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1615999999999997e-05 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002266+0000", 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9477e-05 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002437+0000", 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017133799999999999 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.002483+0000", 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5986999999999999e-05 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.006948+0000", 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044650360000000004 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007036+0000", 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7742000000000001e-05 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.731 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:698 3.b 3:d3cc52a1:::rbd_data.110c17d674e9.000000000000000f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.029002+0000", 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.687820447, 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053900490000000001, 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 698 2024-05-06T00:26:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029002+0000", 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029002+0000", 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029004+0000", 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5960000000000001e-06 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029011+0000", 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9840000000000004e-06 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029031+0000", 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0720000000000002e-05 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029048+0000", 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6192999999999999e-05 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029195+0000", 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000147789 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029408+0000", 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212888 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.029702+0000", 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029400500000000002 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034187+0000", 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044851719999999999 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034228+0000", 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0556000000000003e-05 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034299+0000", 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0733000000000002e-05 2024-05-06T00:26:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034392+0000", 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3412999999999995e-05 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4412.0:697 3.6 e24/18 3:64805dc9:::rbd_data.110c17d674e9.000000000000000e:head v 24'2, mlcod=0'0)", 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.030321+0000", 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6865015339999996, 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038529509999999999, 2024-05-06T00:26:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.030321+0000", 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.030321+0000", 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.030323+0000", 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5999999999999999e-06 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.030337+0000", 2024-05-06T00:26:55.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4229999999999999e-05 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.030364+0000", 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7546999999999999e-05 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.030384+0000", 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9653e-05 2024-05-06T00:26:55.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.030535+0000", 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015151800000000001 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.030574+0000", 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8272999999999998e-05 2024-05-06T00:26:55.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034095+0000", 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035211370000000001 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.034174+0000", 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8992999999999994e-05 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:971 3.19 3:9fda96f1:::rbd_header.110c17d674e9:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.751739+0000", 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9650834110000002, 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040253550000000004, 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 971 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.751739+0000", 2024-05-06T00:26:55.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.751739+0000", 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.751742+0000", 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6730000000000001e-06 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.751754+0000", 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2517000000000001e-05 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.751782+0000", 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7509999999999999e-05 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.751802+0000", 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9987e-05 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.751963+0000", 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016154900000000001 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.751981+0000", 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8094e-05 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.753616+0000", 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001634206 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.753787+0000", 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017155800000000001 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.754416+0000", 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00062904200000000001 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.755553+0000", 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011364490000000001 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.755642+0000", 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9151000000000001e-05 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.755707+0000", 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5563999999999998e-05 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.755764+0000", 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7055000000000002e-05 2024-05-06T00:26:55.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:1079 3.19 3:9fda96f1:::rbd_header.110c17d674e9:head [watch unwatch cookie 140058816813392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.095881+0000", 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.6209417439999996, 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035308200000000001, 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:26:55.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1079 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.095881+0000", 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.095881+0000", 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.095882+0000", 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3200000000000001e-06 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.095889+0000", 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9940000000000003e-06 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.095909+0000", 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9976e-05 2024-05-06T00:26:55.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.095918+0000", 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0170000000000002e-06 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.096704+0000", 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00078548099999999998 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.096834+0000", 2024-05-06T00:26:55.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013081900000000001 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.097122+0000", 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028778000000000001 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.098154+0000", 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001032039 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.099138+0000", 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098434900000000003 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.099218+0000", 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9222999999999997e-05 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.099412+0000", 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019382199999999999 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.756 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:228 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 397312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1198075259999998, 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072743440000000003, 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:55.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 228 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597016+0000", 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.443e-06 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597023+0000", 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8340000000000001e-06 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597044+0000", 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0876000000000001e-05 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597055+0000", 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1314e-05 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597176+0000", 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120565 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597192+0000", 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6337e-05 2024-05-06T00:26:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603743+0000", 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065506820000000004 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603976+0000", 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000233355 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604289+0000", 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031293800000000002 2024-05-06T00:26:55.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:229 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 528384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1196496270000003, 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075584060000000002, 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:26:55.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 229 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597175+0000", 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8929999999999999e-06 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597184+0000", 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2040000000000003e-06 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597200+0000", 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5642999999999999e-05 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597209+0000", 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4329999999999993e-06 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597301+0000", 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1588999999999993e-05 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597309+0000", 2024-05-06T00:26:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604360+0000", 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070513030000000001 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604516+0000", 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015564999999999999 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604731+0000", 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000215267 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4408.0:2225 3.6 e28/18 3:65d72c0c:::rbd_object_map.110c2013632c:head v 28'119, mlcod=27'117)", 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:54.782596+0000", 2024-05-06T00:26:55.767 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.93422641299999998, 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037758929999999998, 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782596+0000", 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782596+0000", 2024-05-06T00:26:55.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782597+0000", 2024-05-06T00:26:55.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0330000000000001e-06 2024-05-06T00:26:55.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782608+0000", 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.046e-05 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782634+0000", 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6152999999999998e-05 2024-05-06T00:26:55.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782653+0000", 2024-05-06T00:26:55.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.95e-05 2024-05-06T00:26:55.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782731+0000", 2024-05-06T00:26:55.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7522999999999996e-05 2024-05-06T00:26:55.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.782753+0000", 2024-05-06T00:26:55.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2433999999999999e-05 2024-05-06T00:26:55.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.786239+0000", 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034858179999999999 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:54.786372+0000", 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000132972 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.775 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:91 3.6 e28/18 3:645d64d7:::rbd_header.110cc8beeb04:head v 28'129, mlcod=28'126)", 2024-05-06T00:26:55.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.608485+0000", 2024-05-06T00:26:55.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.10833747000000001, 2024-05-06T00:26:55.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054914009999999999, 2024-05-06T00:26:55.776 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608485+0000", 2024-05-06T00:26:55.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608485+0000", 2024-05-06T00:26:55.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608486+0000", 2024-05-06T00:26:55.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0759999999999999e-06 2024-05-06T00:26:55.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608497+0000", 2024-05-06T00:26:55.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0801999999999999e-05 2024-05-06T00:26:55.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608510+0000", 2024-05-06T00:26:55.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.331e-05 2024-05-06T00:26:55.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608519+0000", 2024-05-06T00:26:55.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6090000000000001e-06 2024-05-06T00:26:55.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608598+0000", 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9648999999999996e-05 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608649+0000", 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0407e-05 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.613919+0000", 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052703230000000004 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.613976+0000", 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7225000000000002e-05 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:97 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.621892+0000", 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.094930581, 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00388127, 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.784 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 97 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.621892+0000", 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.621892+0000", 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.621893+0000", 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.052e-06 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.621898+0000", 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5330000000000002e-06 2024-05-06T00:26:55.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.787 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:26:55.787 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:26:55.787 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-06T00:26:55.787 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-06T00:26:55.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.621913+0000", 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4538e-05 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.621923+0000", 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0327e-05 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.621983+0000", 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0095e-05 2024-05-06T00:26:55.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.622057+0000", 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3113000000000003e-05 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.622265+0000", 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020808400000000001 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.625388+0000", 2024-05-06T00:26:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003123781 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.625679+0000", 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029096800000000001 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.625723+0000", 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3804999999999998e-05 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.625773+0000", 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9974e-05 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:123 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.648069+0000", 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.068753300000000003, 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042320159999999999, 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 123 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648069+0000", 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648069+0000", 2024-05-06T00:26:55.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648072+0000", 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648084+0000", 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1600000000000001e-05 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648111+0000", 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7321000000000002e-05 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648122+0000", 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0961999999999999e-05 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648197+0000", 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5600999999999997e-05 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648290+0000", 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2641999999999998e-05 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.648611+0000", 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032061399999999999 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.651915+0000", 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033040869999999998 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.652179+0000", 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026460500000000001 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.652245+0000", 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6025999999999994e-05 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.652301+0000", 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5717999999999997e-05 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.798 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:161 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.683279+0000", 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.033543488000000003, 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054288820000000003, 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 161 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683279+0000", 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683279+0000", 2024-05-06T00:26:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683281+0000", 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.486e-06 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683293+0000", 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1404e-05 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683324+0000", 2024-05-06T00:26:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1409999999999999e-05 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683339+0000", 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4513e-05 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683426+0000", 2024-05-06T00:26:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7640999999999997e-05 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683526+0000", 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010002599999999999 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683989+0000", 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00046260000000000002 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687768+0000", 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003778862 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688605+0000", 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00083695299999999998 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688655+0000", 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0179999999999997e-05 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688708+0000", 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2806999999999998e-05 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:162 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.683543+0000", 2024-05-06T00:26:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.033279014000000003, 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053251959999999999, 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 162 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683543+0000", 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683543+0000", 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683545+0000", 2024-05-06T00:26:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.404e-06 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683552+0000", 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.503e-06 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683581+0000", 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8779999999999999e-05 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683596+0000", 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.505e-05 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684548+0000", 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095151199999999995 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684811+0000", 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026319800000000002 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.685887+0000", 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010756120000000001 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688718+0000", 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028318089999999998 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688783+0000", 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4906999999999994e-05 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688833+0000", 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9607999999999997e-05 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688869+0000", 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5812999999999997e-05 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:163 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.033194998000000003, 2024-05-06T00:26:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060279510000000001, 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 163 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:26:55.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683629+0000", 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.28e-06 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683636+0000", 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2729999999999997e-06 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683662+0000", 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5754000000000001e-05 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683673+0000", 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0890000000000001e-05 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.686748+0000", 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030753180000000001 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.686848+0000", 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9697000000000004e-05 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687256+0000", 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040807300000000001 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688723+0000", 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014670740000000001 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689566+0000", 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00084270500000000002 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689616+0000", 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0358999999999997e-05 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689655+0000", 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9527999999999998e-05 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.818 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:165 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.032665884999999999, 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010266747, 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 165 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684158+0000", 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6929999999999999e-06 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684170+0000", 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1933000000000001e-05 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684198+0000", 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8266999999999999e-05 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684221+0000", 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2736999999999999e-05 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687824+0000", 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003602863 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687915+0000", 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0804999999999995e-05 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688264+0000", 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034917100000000001 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693979+0000", 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057148959999999997 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694148+0000", 2024-05-06T00:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168951 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694315+0000", 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000167016 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694423+0000", 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108415 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:172 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.689594+0000", 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.027228760000000001, 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051283839999999997, 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 172 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.825 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689594+0000", 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689594+0000", 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689595+0000", 2024-05-06T00:26:55.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6330000000000001e-06 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689604+0000", 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5439999999999999e-06 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689623+0000", 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8753000000000002e-05 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689635+0000", 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1956e-05 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689749+0000", 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011414899999999999 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689920+0000", 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000171366 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690883+0000", 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096281600000000002 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694520+0000", 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003636809 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694555+0000", 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5064000000000002e-05 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694662+0000", 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010694800000000001 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694722+0000", 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0346000000000001e-05 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:173 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.027155400999999999, 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058174089999999999, 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 173 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689668+0000", 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2669999999999999e-06 2024-05-06T00:26:55.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689676+0000", 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8569999999999995e-06 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689696+0000", 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0006000000000001e-05 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689711+0000", 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-06T00:26:55.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.691939+0000", 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002227559 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.692171+0000", 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000232475 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:55.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693111+0000", 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000939795 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694929+0000", 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001817281 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695267+0000", 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033894600000000002 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695385+0000", 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117683 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695484+0000", 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9343000000000001e-05 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:174 3.19 3:9fbbd60a:::rbd_data.110cc8beeb04.000000000000001b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1431552~10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.689984+0000", 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.026838549999999999, 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036798149999999999, 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 174 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689984+0000", 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689984+0000", 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689986+0000", 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9769999999999999e-06 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690013+0000", 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7475999999999999e-05 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690037+0000", 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.321e-05 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690060+0000", 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3737e-05 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690140+0000", 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9783000000000005e-05 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690270+0000", 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012976700000000001 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690466+0000", 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019649000000000001 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.692174+0000", 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001707447 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693469+0000", 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001295196 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693555+0000", 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6422999999999996e-05 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693664+0000", 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108309 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4578.0:165 3.4 e28/18 ondisk, result = 0)", 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.690258+0000", 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.026564028999999999, 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041860980000000001, 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690258+0000", 2024-05-06T00:26:55.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690258+0000", 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690260+0000", 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1969999999999999e-06 2024-05-06T00:26:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690268+0000", 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6270000000000006e-06 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690289+0000", 2024-05-06T00:26:55.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1172999999999999e-05 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.690304+0000", 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.414e-05 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:55.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694108+0000", 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003804347 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694130+0000", 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1783000000000001e-05 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694445+0000", 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031483100000000001 2024-05-06T00:26:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.849 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:55.849 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:55.849 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:55.850 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:26:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:26:55.852 INFO:teuthology.orchestra.run.smithi186.stdout:354 write 0xa1d200 thru 0xa22d50 (0x5b51 bytes) 2024-05-06T00:26:55.852 INFO:teuthology.orchestra.run.smithi186.stdout:356 write 0x664da00 thru 0x664f121 (0x1722 bytes) 2024-05-06T00:26:55.852 INFO:teuthology.orchestra.run.smithi186.stdout:360 read 0x1d33a00 thru 0x1d4390b (0xff0c bytes) 2024-05-06T00:26:55.853 INFO:teuthology.orchestra.run.smithi186.stdout:366 read 0x7c48200 thru 0x7c56148 (0xdf49 bytes) 2024-05-06T00:26:55.853 INFO:teuthology.orchestra.run.smithi186.stdout:368 read 0x4b11600 thru 0x4b1fee1 (0xe8e2 bytes) 2024-05-06T00:26:55.853 INFO:teuthology.orchestra.run.smithi186.stdout:371 read 0x9e49e00 thru 0x9e4eb81 (0x4d82 bytes) 2024-05-06T00:26:55.853 INFO:teuthology.orchestra.run.smithi186.stdout:373 read 0x3f97a00 thru 0x3fa1537 (0x9b38 bytes) 2024-05-06T00:26:55.854 INFO:teuthology.orchestra.run.smithi186.stdout:375 write 0x4ee1e00 thru 0x4eebe66 (0xa067 bytes) 2024-05-06T00:26:55.855 INFO:teuthology.orchestra.run.smithi186.stdout:376 write 0x71c3600 thru 0x71c526a (0x1c6b bytes) 2024-05-06T00:26:55.855 INFO:teuthology.orchestra.run.smithi186.stdout:377 punch from 0x9c2a600 to 0x9c2fe00, (0x5800 bytes) 2024-05-06T00:26:55.855 INFO:teuthology.orchestra.run.smithi186.stdout:381 trunc from 0xbbcf400 to 0xdca00 2024-05-06T00:26:55.856 INFO:teuthology.orchestra.run.smithi186.stdout:384 read 0x59e00 thru 0x5d2a8 (0x34a9 bytes) 2024-05-06T00:26:55.856 INFO:teuthology.orchestra.run.smithi186.stdout:385 read 0x7c200 thru 0x8000e (0x3e0f bytes) 2024-05-06T00:26:55.857 INFO:teuthology.orchestra.run.smithi186.stdout:386 trunc from 0xdca00 to 0x4546000 2024-05-06T00:26:55.870 INFO:teuthology.orchestra.run.smithi186.stdout:387 punch from 0x9b8400 to 0x9bc600, (0x4200 bytes) 2024-05-06T00:26:55.900 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:55.897+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:55.912 INFO:teuthology.orchestra.run.smithi186.stdout:390 write 0xdd12800 thru 0xdd17bcd (0x53ce bytes) 2024-05-06T00:26:55.914 INFO:teuthology.orchestra.run.smithi186.stdout:391 read 0x8534c00 thru 0x8534c41 (0x42 bytes) 2024-05-06T00:26:55.914 INFO:teuthology.orchestra.run.smithi186.stdout:392 read 0x3945600 thru 0x3947213 (0x1c14 bytes) 2024-05-06T00:26:55.914 INFO:teuthology.orchestra.run.smithi186.stdout:394 write 0xb94fc00 thru 0xb955aee (0x5eef bytes) 2024-05-06T00:26:55.914 INFO:teuthology.orchestra.run.smithi186.stdout:398 read 0x3daa600 thru 0x3dae04d (0x3a4e bytes) 2024-05-06T00:26:55.943 INFO:teuthology.orchestra.run.smithi186.stdout:399 write 0xe24ea00 thru 0xe254972 (0x5f73 bytes) 2024-05-06T00:26:55.944 INFO:teuthology.orchestra.run.smithi186.stdout:400 read 0x44c9e00 thru 0x44cd903 (0x3b04 bytes) 2024-05-06T00:26:55.945 INFO:teuthology.orchestra.run.smithi186.stdout:401 trunc from 0xe254a00 to 0xd883600 2024-05-06T00:26:55.954 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:55.953+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:55.977 INFO:teuthology.orchestra.run.smithi186.stdout:402 write 0xa2b0200 thru 0xa2b06f6 (0x4f7 bytes) 2024-05-06T00:26:55.996 INFO:teuthology.orchestra.run.smithi186.stdout:404 trunc from 0xd883600 to 0x4b5c400 2024-05-06T00:26:56.026 INFO:teuthology.orchestra.run.smithi186.stdout:405 write 0x1fe3800 thru 0x1fe599c (0x219d bytes) 2024-05-06T00:26:56.028 INFO:teuthology.orchestra.run.smithi186.stdout:406 read 0x1051e00 thru 0x1059044 (0x7245 bytes) 2024-05-06T00:26:56.034 INFO:teuthology.orchestra.run.smithi186.stdout:408 trunc from 0x4b5c400 to 0x31e8600 2024-05-06T00:26:56.039 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:56.040 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:56.055 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:56.053+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:56.059 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:26:56.076 INFO:teuthology.orchestra.run.smithi186.stdout:410 trunc from 0x31e8600 to 0xae68200 2024-05-06T00:26:56.088 INFO:teuthology.orchestra.run.smithi186.stdout:412 punch from 0x7c3a600 to 0x7c3f400, (0x4e00 bytes) 2024-05-06T00:26:56.090 INFO:teuthology.orchestra.run.smithi186.stdout:414 read 0x5958e00 thru 0x5965860 (0xca61 bytes) 2024-05-06T00:26:56.090 INFO:teuthology.orchestra.run.smithi186.stdout:415 punch from 0x10c7800 to 0x10c8c00, (0x1400 bytes) 2024-05-06T00:26:56.091 INFO:teuthology.orchestra.run.smithi186.stdout:416 write 0xa19e200 thru 0xa1a0e23 (0x2c24 bytes) 2024-05-06T00:26:56.109 INFO:teuthology.orchestra.run.smithi186.stdout:418 write 0xb159000 thru 0xb167baf (0xebb0 bytes) 2024-05-06T00:26:56.111 INFO:teuthology.orchestra.run.smithi186.stdout:419 write 0x8c52800 thru 0x8c57a8c (0x528d bytes) 2024-05-06T00:26:56.115 INFO:teuthology.orchestra.run.smithi186.stdout:422 trunc from 0xb167c00 to 0x8db9200 2024-05-06T00:26:56.148 INFO:teuthology.orchestra.run.smithi186.stdout:425 punch from 0x243de00 to 0x2446a00, (0x8c00 bytes) 2024-05-06T00:26:56.149 INFO:teuthology.orchestra.run.smithi186.stdout:426 punch from 0x7cd5400 to 0x7cdbe00, (0x6a00 bytes) 2024-05-06T00:26:56.155 INFO:teuthology.orchestra.run.smithi186.stdout:428 trunc from 0x8db9200 to 0xb184c00 2024-05-06T00:26:56.156 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:56.157+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:56.167 INFO:teuthology.orchestra.run.smithi186.stdout:429 punch from 0x16f9200 to 0x1707e00, (0xec00 bytes) 2024-05-06T00:26:56.168 INFO:teuthology.orchestra.run.smithi186.stdout:430 read 0xb085a00 thru 0xb089c5c (0x425d bytes) 2024-05-06T00:26:56.168 INFO:teuthology.orchestra.run.smithi186.stdout:431 write 0x54f600 thru 0x552182 (0x2b83 bytes) 2024-05-06T00:26:56.169 INFO:teuthology.orchestra.run.smithi186.stdout:432 read 0xe62a00 thru 0xe699fd (0x6ffe bytes) 2024-05-06T00:26:56.170 INFO:teuthology.orchestra.run.smithi186.stdout:433 write 0x376fc00 thru 0x3775368 (0x5769 bytes) 2024-05-06T00:26:56.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:26:56.193 INFO:teuthology.orchestra.run.smithi186.stdout:435 write 0xda21a00 thru 0xda23011 (0x1612 bytes) 2024-05-06T00:26:56.195 INFO:teuthology.orchestra.run.smithi186.stdout:436 punch from 0x6c79a00 to 0x6c7d400, (0x3a00 bytes) 2024-05-06T00:26:56.195 INFO:teuthology.orchestra.run.smithi186.stdout:437 read 0xa131a00 thru 0xa13679f (0x4da0 bytes) 2024-05-06T00:26:56.195 INFO:teuthology.orchestra.run.smithi186.stdout:440 read 0x90dc400 thru 0x90e4496 (0x8097 bytes) 2024-05-06T00:26:56.196 INFO:teuthology.orchestra.run.smithi186.stdout:441 read 0x2482800 thru 0x2487f4d (0x574e bytes) 2024-05-06T00:26:56.196 INFO:teuthology.orchestra.run.smithi186.stdout:442 read 0x70ca000 thru 0x70d6e3a (0xce3b bytes) 2024-05-06T00:26:56.197 INFO:teuthology.orchestra.run.smithi186.stdout:443 read 0x7339200 thru 0x7345563 (0xc364 bytes) 2024-05-06T00:26:56.217 INFO:teuthology.orchestra.run.smithi186.stdout:444 write 0xe887000 thru 0xe8871a3 (0x1a4 bytes) 2024-05-06T00:26:56.218 INFO:teuthology.orchestra.run.smithi186.stdout:445 write 0xdb68800 thru 0xdb68bea (0x3eb bytes) 2024-05-06T00:26:56.234 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:56.235 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:56.235 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:56.235 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:56.235 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:56.238 INFO:teuthology.orchestra.run.smithi186.stdout:446 trunc from 0xe887200 to 0x1804200 2024-05-06T00:26:56.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:26:56.261 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:56.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:56.261 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 29, 2024-05-06T00:26:56.261 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b583c614-b2ca-4769-90c0-44726bf69b34", 2024-05-06T00:26:56.261 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-06T00:26:16.923985+0000", 2024-05-06T00:26:56.261 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-06T00:26:55.759493+0000", 2024-05-06T00:26:56.261 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-06T00:26:27.394947+0000", 2024-05-06T00:26:56.261 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-06T00:26:18.750504+0000", 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5801984, 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "noscrub", 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-06T00:26:56.262 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 6, 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "luminous", 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:28.436494+0000", 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-06T00:26:56.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "14", 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:26:56.266 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:26:56.267 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-06T00:26:56.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 6.059999942779541, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 6.059999942779541, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.33000001311302185, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 1, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 1 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:33.941780+0000", 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:26:56.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-06T00:26:56.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:26:56.272 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:26:56.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:26:56.274 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.25, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.25, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 1, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2.25, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2.25, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 1, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 1 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:42.622595+0000", 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-06T00:26:56.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-06T00:26:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "28", 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:26:56.277 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 5, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 28, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:26:56.278 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:26:56.279 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 1.5, 2024-05-06T00:26:56.280 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 1.5, 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 1, 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.5, 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.5, 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 1, 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 1 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d31b9a6a-864b-4c24-99db-8cc6cf3a5c24", 2024-05-06T00:26:56.281 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6817/1427564096", 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6819/1427564096", 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6823/1427564096", 2024-05-06T00:26:56.286 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6821/1427564096", 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "86509826-0e53-4678-8260-17c97c05af54", 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:56.287 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.289 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.291 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6803/2663808859", 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6807/2663808859", 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/2663808859", 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6811/2663808859", 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.292 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "8973b524-521e-41c0-87f2-85cb390fbfeb", 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:56.296 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6802/385938386", 2024-05-06T00:26:56.297 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6806/385938386", 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6814/385938386", 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6810/385938386", 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "3b0d07bd-132e-48ff-83ac-7182eee05df9", 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:56.298 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6811", 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6812", 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6814", 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6816", 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6821", 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6822", 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6817", 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6819", 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6812/4178038317", 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6816/4178038317", 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6822/4178038317", 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6819/4178038317", 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:56.303 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2", 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:56.304 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 25, 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6808", 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6809", 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6810", 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:56.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6813", 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6820", 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6823", 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6815", 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6818", 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6809/1052555350", 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6813/1052555350", 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1052555350", 2024-05-06T00:26:56.309 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6818/1052555350", 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "34b61f5f-e407-4414-8bc5-11263bbe261a", 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:26:56.310 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 18, 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6800", 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.311 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6801", 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6802", 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6803", 2024-05-06T00:26:56.312 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6806", 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6807", 2024-05-06T00:26:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6804", 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6805", 2024-05-06T00:26:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6801/2019146845", 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6803/2019146845", 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6807/2019146845", 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6805/2019146845", 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-06T00:26:56.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.276988+0000", 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:56.316 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.054473+0000", 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.154558+0000", 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.284746+0000", 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.259847+0000", 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.180375+0000", 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:26:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [ 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.15", 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: 4, 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [], 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [], 2024-05-06T00:26:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [], 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [], 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-06T00:26:56.321 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:56.322 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:56.322 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:56.257+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:56.325 INFO:tasks.thrashosds.thrasher:j is {'epoch': 29, 'fsid': 'b583c614-b2ca-4769-90c0-44726bf69b34', 'created': '2024-05-06T00:26:16.923985+0000', 'modified': '2024-05-06T00:26:55.759493+0000', 'last_up_change': '2024-05-06T00:26:27.394947+0000', 'last_in_change': '2024-05-06T00:26:18.750504+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T00:26:28.436494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-05-06T00:26:33.941780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-06T00:26:42.622595+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd31b9a6a-864b-4c24-99db-8cc6cf3a5c24', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6817', 'nonce': 1427564096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6818', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 1427564096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6822', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 1427564096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6821', 'nonce': 1427564096}]}, 'public_addr': '172.21.15.78:6817/1427564096', 'cluster_addr': '172.21.15.78:6819/1427564096', 'heartbeat_back_addr': '172.21.15.78:6823/1427564096', 'heartbeat_front_addr': '172.21.15.78:6821/1427564096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '86509826-0e53-4678-8260-17c97c05af54', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6801', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6803', 'nonce': 2663808859}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6805', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6807', 'nonce': 2663808859}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6813', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 2663808859}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6809', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6811', 'nonce': 2663808859}]}, 'public_addr': '172.21.15.78:6803/2663808859', 'cluster_addr': '172.21.15.78:6807/2663808859', 'heartbeat_back_addr': '172.21.15.78:6815/2663808859', 'heartbeat_front_addr': '172.21.15.78:6811/2663808859', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8973b524-521e-41c0-87f2-85cb390fbfeb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6802', 'nonce': 385938386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6804', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6806', 'nonce': 385938386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6812', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6814', 'nonce': 385938386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6808', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6810', 'nonce': 385938386}]}, 'public_addr': '172.21.15.78:6802/385938386', 'cluster_addr': '172.21.15.78:6806/385938386', 'heartbeat_back_addr': '172.21.15.78:6814/385938386', 'heartbeat_front_addr': '172.21.15.78:6810/385938386', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3b0d07bd-132e-48ff-83ac-7182eee05df9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6811', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6812', 'nonce': 4178038317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6816', 'nonce': 4178038317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6821', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6822', 'nonce': 4178038317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6817', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 4178038317}]}, 'public_addr': '172.21.15.146:6812/4178038317', 'cluster_addr': '172.21.15.146:6816/4178038317', 'heartbeat_back_addr': '172.21.15.146:6822/4178038317', 'heartbeat_front_addr': '172.21.15.146:6819/4178038317', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 1052555350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 1052555350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1052555350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6815', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6818', 'nonce': 1052555350}]}, 'public_addr': '172.21.15.146:6809/1052555350', 'cluster_addr': '172.21.15.146:6813/1052555350', 'heartbeat_back_addr': '172.21.15.146:6823/1052555350', 'heartbeat_front_addr': '172.21.15.146:6818/1052555350', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '34b61f5f-e407-4414-8bc5-11263bbe261a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 2019146845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 2019146845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 2019146845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 2019146845}]}, 'public_addr': '172.21.15.146:6801/2019146845', 'cluster_addr': '172.21.15.146:6803/2019146845', 'heartbeat_back_addr': '172.21.15.146:6807/2019146845', 'heartbeat_front_addr': '172.21.15.146:6805/2019146845', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.276988+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.054473+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.154558+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.284746+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.259847+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.180375+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.15', 'osds': [4, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-06T00:26:56.326 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:26:56.326 INFO:teuthology.orchestra.run.smithi186.stdout:449 trunc from 0x1804200 to 0x60a4c00 2024-05-06T00:26:56.326 INFO:teuthology.orchestra.run.smithi186.stdout:450 read 0xe61400 thru 0xe6ab90 (0x9791 bytes) 2024-05-06T00:26:56.340 INFO:teuthology.orchestra.run.smithi186.stdout:451 write 0xd2fae00 thru 0xd2fc749 (0x194a bytes) 2024-05-06T00:26:56.346 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:26:56.347 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:26:56.366 INFO:teuthology.orchestra.run.smithi186.stdout:452 write 0xe1d8e00 thru 0xe1e3197 (0xa398 bytes) 2024-05-06T00:26:56.368 INFO:teuthology.orchestra.run.smithi186.stdout:453 read 0x6fbea00 thru 0x6fc874c (0x9d4d bytes) 2024-05-06T00:26:56.368 INFO:teuthology.orchestra.run.smithi186.stdout:454 read 0x7d10400 thru 0x7d1ecce (0xe8cf bytes) 2024-05-06T00:26:56.368 INFO:teuthology.orchestra.run.smithi186.stdout:455 write 0xd77c600 thru 0xd789e1c (0xd81d bytes) 2024-05-06T00:26:56.368 INFO:teuthology.orchestra.run.smithi186.stdout:457 punch from 0x1434800 to 0x1440c00, (0xc400 bytes) 2024-05-06T00:26:56.369 INFO:teuthology.orchestra.run.smithi186.stdout:458 read 0x718fe00 thru 0x719517e (0x537f bytes) 2024-05-06T00:26:56.369 INFO:teuthology.orchestra.run.smithi186.stdout:459 read 0xc3af200 thru 0xc3af2ff (0x100 bytes) 2024-05-06T00:26:56.369 INFO:teuthology.orchestra.run.smithi186.stdout:460 read 0x192f600 thru 0x19379da (0x83db bytes) 2024-05-06T00:26:56.370 INFO:teuthology.orchestra.run.smithi186.stdout:463 read 0xd413a00 thru 0xd41614b (0x274c bytes) 2024-05-06T00:26:56.370 INFO:teuthology.orchestra.run.smithi186.stdout:464 write 0x8d9e00 thru 0x8e6999 (0xcb9a bytes) 2024-05-06T00:26:56.371 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:26:56.371 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:26:56.371 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:56.369+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:56.374 INFO:teuthology.orchestra.run.smithi186.stdout:466 punch from 0x4a95600 to 0x4a96c00, (0x1600 bytes) 2024-05-06T00:26:56.376 INFO:teuthology.orchestra.run.smithi186.stdout:467 punch from 0x1864a00 to 0x186de00, (0x9400 bytes) 2024-05-06T00:26:56.376 INFO:teuthology.orchestra.run.smithi186.stdout:470 punch from 0x36d6600 to 0x36e2800, (0xc200 bytes) 2024-05-06T00:26:56.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:26:56.393 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:26:56.394 INFO:teuthology.orchestra.run.smithi186.stdout:472 trunc from 0xe1e3200 to 0x3f2ac00 2024-05-06T00:26:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:56.405 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:26:56.418 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:26:56.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:26:56.423 INFO:teuthology.orchestra.run.smithi186.stdout:473 write 0x876c00 thru 0x878730 (0x1b31 bytes) 2024-05-06T00:26:56.427 INFO:teuthology.orchestra.run.smithi186.stdout:474 punch from 0x37eea00 to 0x37f2000, (0x3600 bytes) 2024-05-06T00:26:56.439 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:26:56.439 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:26:56.450 INFO:teuthology.orchestra.run.smithi186.stdout:476 write 0x9712a00 thru 0x9721894 (0xee95 bytes) 2024-05-06T00:26:56.451 INFO:teuthology.orchestra.run.smithi186.stdout:479 punch from 0x8913600 to 0x8920000, (0xca00 bytes) 2024-05-06T00:26:56.451 INFO:teuthology.orchestra.run.smithi186.stdout:481 write 0x1af7600 thru 0x1b02a20 (0xb421 bytes) 2024-05-06T00:26:56.463 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:26:56.463 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:26:56.468 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:56.469+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:56.476 INFO:teuthology.orchestra.run.smithi186.stdout:482 write 0xab27e00 thru 0xab28830 (0xa31 bytes) 2024-05-06T00:26:56.477 INFO:teuthology.orchestra.run.smithi186.stdout:484 write 0x1fd3a00 thru 0x1fe1d76 (0xe377 bytes) 2024-05-06T00:26:56.477 INFO:teuthology.orchestra.run.smithi186.stdout:485 read 0x46fc200 thru 0x4704652 (0x8453 bytes) 2024-05-06T00:26:56.478 INFO:teuthology.orchestra.run.smithi186.stdout:488 read 0x93ebe00 thru 0x93f83d5 (0xc5d6 bytes) 2024-05-06T00:26:56.478 INFO:teuthology.orchestra.run.smithi186.stdout:489 read 0x22d6e00 thru 0x22e4392 (0xd593 bytes) 2024-05-06T00:26:56.482 INFO:teuthology.orchestra.run.smithi186.stdout:490 read 0x9b35000 thru 0x9b43cb3 (0xecb4 bytes) 2024-05-06T00:26:56.482 INFO:teuthology.orchestra.run.smithi186.stdout:492 write 0xa226a00 thru 0xa227997 (0xf98 bytes) 2024-05-06T00:26:56.482 INFO:teuthology.orchestra.run.smithi186.stdout:493 read 0x553dc00 thru 0x55485b2 (0xa9b3 bytes) 2024-05-06T00:26:56.482 INFO:teuthology.orchestra.run.smithi186.stdout:494 read 0x9a5c200 thru 0x9a65904 (0x9705 bytes) 2024-05-06T00:26:56.483 INFO:teuthology.orchestra.run.smithi186.stdout:495 write 0x1019c00 thru 0x1020a81 (0x6e82 bytes) 2024-05-06T00:26:56.483 INFO:teuthology.orchestra.run.smithi186.stdout:496 punch from 0x7980200 to 0x798c200, (0xc000 bytes) 2024-05-06T00:26:56.489 INFO:teuthology.orchestra.run.smithi186.stdout:498 trunc from 0xab28a00 to 0xd8aee00 2024-05-06T00:26:56.505 INFO:teuthology.orchestra.run.smithi186.stdout:501 read 0x4237600 thru 0x4246db1 (0xf7b2 bytes) 2024-05-06T00:26:56.506 INFO:teuthology.orchestra.run.smithi186.stdout:502 write 0x30ada00 thru 0x30b8b08 (0xb109 bytes) 2024-05-06T00:26:56.508 INFO:teuthology.orchestra.run.smithi186.stdout:505 read 0x867be00 thru 0x867ff37 (0x4138 bytes) 2024-05-06T00:26:56.509 INFO:teuthology.orchestra.run.smithi186.stdout:506 punch from 0xb821e00 to 0xb830e00, (0xf000 bytes) 2024-05-06T00:26:56.509 INFO:teuthology.orchestra.run.smithi186.stdout:508 read 0x3721600 thru 0x372f3c5 (0xddc6 bytes) 2024-05-06T00:26:56.510 INFO:teuthology.orchestra.run.smithi186.stdout:510 write 0x15c5200 thru 0x15d423c (0xf03d bytes) 2024-05-06T00:26:56.510 INFO:teuthology.orchestra.run.smithi186.stdout:512 read 0x2feb800 thru 0x2fee49d (0x2c9e bytes) 2024-05-06T00:26:56.511 INFO:teuthology.orchestra.run.smithi186.stdout:514 write 0xb9ce00 thru 0xba37bb (0x69bc bytes) 2024-05-06T00:26:56.513 INFO:teuthology.orchestra.run.smithi186.stdout:515 read 0x365d200 thru 0x365e093 (0xe94 bytes) 2024-05-06T00:26:56.513 INFO:teuthology.orchestra.run.smithi186.stdout:516 read 0x4f97800 thru 0x4f9e830 (0x7031 bytes) 2024-05-06T00:26:56.513 INFO:teuthology.orchestra.run.smithi186.stdout:519 read 0x52a0400 thru 0x52a6b38 (0x6739 bytes) 2024-05-06T00:26:56.513 INFO:teuthology.orchestra.run.smithi186.stdout:521 read 0x64afe00 thru 0x64baaae (0xacaf bytes) 2024-05-06T00:26:56.513 INFO:teuthology.orchestra.run.smithi186.stdout:524 read 0xae22a00 thru 0xae2666c (0x3c6d bytes) 2024-05-06T00:26:56.513 INFO:teuthology.orchestra.run.smithi186.stdout:526 read 0x5084000 thru 0x5090c74 (0xcc75 bytes) 2024-05-06T00:26:56.514 INFO:teuthology.orchestra.run.smithi186.stdout:529 write 0xa719600 thru 0xa7205d6 (0x6fd7 bytes) 2024-05-06T00:26:56.523 INFO:teuthology.orchestra.run.smithi186.stdout:532 trunc from 0xd8aee00 to 0x8a1fe00 2024-05-06T00:26:56.557 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:56.557 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:56.559 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:56.557+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:56.562 INFO:teuthology.orchestra.run.smithi186.stdout:533 write 0x8799a00 thru 0x879b23c (0x183d bytes) 2024-05-06T00:26:56.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:26:56.572 INFO:teuthology.orchestra.run.smithi186.stdout:536 trunc from 0x8a1fe00 to 0xae9b800 2024-05-06T00:26:56.588 INFO:teuthology.orchestra.run.smithi186.stdout:537 read 0x1f93c00 thru 0x1f9f217 (0xb618 bytes) 2024-05-06T00:26:56.591 INFO:teuthology.orchestra.run.smithi186.stdout:539 read 0x4a5cc00 thru 0x4a61af1 (0x4ef2 bytes) 2024-05-06T00:26:56.608 INFO:teuthology.orchestra.run.smithi186.stdout:541 trunc from 0xae9b800 to 0x1bebe00 2024-05-06T00:26:56.635 INFO:teuthology.orchestra.run.smithi186.stdout:542 punch from 0x1888800 to 0x1895c00, (0xd400 bytes) 2024-05-06T00:26:56.635 INFO:teuthology.orchestra.run.smithi186.stdout:545 punch from 0x1ad5200 to 0x1addc00, (0x8a00 bytes) 2024-05-06T00:26:56.660 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:56.661+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:56.663 INFO:teuthology.orchestra.run.smithi186.stdout:547 write 0x5b42800 thru 0x5b460ad (0x38ae bytes) 2024-05-06T00:26:56.665 INFO:teuthology.orchestra.run.smithi186.stdout:549 read 0x53b7400 thru 0x53c07fe (0x93ff bytes) 2024-05-06T00:26:56.692 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:26:56.693 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:26:56.698 INFO:teuthology.orchestra.run.smithi186.stdout:550 write 0xb4cc200 thru 0xb4d019b (0x3f9c bytes) 2024-05-06T00:26:56.710 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-05-06T00:26:56.400269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":184017701,"num_objects":42,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":6,"num_read":4099,"num_read_kb":8352,"num_write":451,"num_write_kb":3691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":174056,"kb_used_data":3620,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566056984,"statfs":{"total":579820584960,"available":579642351616,"internally_reserved":0,"allocated":3706880,"data_stored":2548314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":174447264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":183427268,"num_objects":35,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":6,"num_read":3969,"num_read_kb":8246,"num_write":344,"num_write_kb":2057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.802346"},"pg_stats":[{"pgid":"3.1f","version":"26'4","reported_seq":47,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.287950+0000","last_change":"2024-05-06T00:26:45.712771+0000","last_active":"2024-05-06T00:26:52.287950+0000","last_peered":"2024-05-06T00:26:52.287950+0000","last_clean":"2024-05-06T00:26:52.287950+0000","last_became_active":"2024-05-06T00:26:43.694494+0000","last_became_peered":"2024-05-06T00:26:43.694494+0000","last_unstale":"2024-05-06T00:26:52.287950+0000","last_undegraded":"2024-05-06T00:26:52.287950+0000","last_fullsized":"2024-05-06T00:26:52.287950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:10:19.351300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":2860086,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":8,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.601802+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:26:51.601802+0000","last_peered":"2024-05-06T00:26:51.601802+0000","last_clean":"2024-05-06T00:26:51.601802+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:26:51.601802+0000","last_undegraded":"2024-05-06T00:26:51.601802+0000","last_fullsized":"2024-05-06T00:26:51.601802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:04:31.047808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.684977+0000","last_change":"2024-05-06T00:26:45.733037+0000","last_active":"2024-05-06T00:26:51.684977+0000","last_peered":"2024-05-06T00:26:51.684977+0000","last_clean":"2024-05-06T00:26:51.684977+0000","last_became_active":"2024-05-06T00:26:43.711475+0000","last_became_peered":"2024-05-06T00:26:43.711475+0000","last_unstale":"2024-05-06T00:26:51.684977+0000","last_undegraded":"2024-05-06T00:26:51.684977+0000","last_fullsized":"2024-05-06T00:26:51.684977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:50:50.319697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"25'3","reported_seq":45,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.684552+0000","last_change":"2024-05-06T00:26:45.732216+0000","last_active":"2024-05-06T00:26:51.684552+0000","last_peered":"2024-05-06T00:26:51.684552+0000","last_clean":"2024-05-06T00:26:51.684552+0000","last_became_active":"2024-05-06T00:26:43.715832+0000","last_became_peered":"2024-05-06T00:26:43.715832+0000","last_unstale":"2024-05-06T00:26:51.684552+0000","last_undegraded":"2024-05-06T00:26:51.684552+0000","last_fullsized":"2024-05-06T00:26:51.684552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:28:35.185887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"25'92","reported_seq":490,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.484030+0000","last_change":"2024-05-06T00:26:45.713563+0000","last_active":"2024-05-06T00:26:52.484030+0000","last_peered":"2024-05-06T00:26:52.484030+0000","last_clean":"2024-05-06T00:26:52.484030+0000","last_became_active":"2024-05-06T00:26:43.707253+0000","last_became_peered":"2024-05-06T00:26:43.707253+0000","last_unstale":"2024-05-06T00:26:52.484030+0000","last_undegraded":"2024-05-06T00:26:52.484030+0000","last_fullsized":"2024-05-06T00:26:52.484030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:40:00.780983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":20971555,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":758,"num_read_kb":2575,"num_write":144,"num_write_kb":210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"26'16","reported_seq":206,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.492098+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:26:52.492098+0000","last_peered":"2024-05-06T00:26:52.492098+0000","last_clean":"2024-05-06T00:26:52.492098+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:26:52.492098+0000","last_undegraded":"2024-05-06T00:26:52.492098+0000","last_fullsized":"2024-05-06T00:26:52.492098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:52:33.023301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":18354195,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":163,"num_read_kb":375,"num_write":12,"num_write_kb":217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"26'50","reported_seq":659,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.435449+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:26:52.435449+0000","last_peered":"2024-05-06T00:26:52.435449+0000","last_clean":"2024-05-06T00:26:52.435449+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:26:52.435449+0000","last_undegraded":"2024-05-06T00:26:52.435449+0000","last_fullsized":"2024-05-06T00:26:52.435449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:14:00.776729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":11343360,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1200,"num_read_kb":1122,"num_write":31,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"23'2","reported_seq":56,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.600604+0000","last_change":"2024-05-06T00:26:45.713666+0000","last_active":"2024-05-06T00:26:51.600604+0000","last_peered":"2024-05-06T00:26:51.600604+0000","last_clean":"2024-05-06T00:26:51.600604+0000","last_became_active":"2024-05-06T00:26:43.688284+0000","last_became_peered":"2024-05-06T00:26:43.688284+0000","last_unstale":"2024-05-06T00:26:51.600604+0000","last_undegraded":"2024-05-06T00:26:51.600604+0000","last_fullsized":"2024-05-06T00:26:51.600604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:08:31.113042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":141,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"26'37","reported_seq":137,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.451568+0000","last_change":"2024-05-06T00:26:45.713245+0000","last_active":"2024-05-06T00:26:52.451568+0000","last_peered":"2024-05-06T00:26:52.451568+0000","last_clean":"2024-05-06T00:26:52.451568+0000","last_became_active":"2024-05-06T00:26:43.698443+0000","last_became_peered":"2024-05-06T00:26:43.698443+0000","last_unstale":"2024-05-06T00:26:52.451568+0000","last_undegraded":"2024-05-06T00:26:52.451568+0000","last_fullsized":"2024-05-06T00:26:52.451568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:02:42.316241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":27791402,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":130,"num_read_kb":342,"num_write":37,"num_write_kb":269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.685018+0000","last_change":"2024-05-06T00:26:45.733718+0000","last_active":"2024-05-06T00:26:51.685018+0000","last_peered":"2024-05-06T00:26:51.685018+0000","last_clean":"2024-05-06T00:26:51.685018+0000","last_became_active":"2024-05-06T00:26:43.703465+0000","last_became_peered":"2024-05-06T00:26:43.703465+0000","last_unstale":"2024-05-06T00:26:51.685018+0000","last_undegraded":"2024-05-06T00:26:51.685018+0000","last_fullsized":"2024-05-06T00:26:51.685018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:50:37.631755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.612646+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:26:51.612646+0000","last_peered":"2024-05-06T00:26:51.612646+0000","last_clean":"2024-05-06T00:26:51.612646+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:26:51.612646+0000","last_undegraded":"2024-05-06T00:26:51.612646+0000","last_fullsized":"2024-05-06T00:26:51.612646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:46:23.885187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.601148+0000","last_change":"2024-05-06T00:26:45.710285+0000","last_active":"2024-05-06T00:26:51.601148+0000","last_peered":"2024-05-06T00:26:51.601148+0000","last_clean":"2024-05-06T00:26:51.601148+0000","last_became_active":"2024-05-06T00:26:43.706306+0000","last_became_peered":"2024-05-06T00:26:43.706306+0000","last_unstale":"2024-05-06T00:26:51.601148+0000","last_undegraded":"2024-05-06T00:26:51.601148+0000","last_fullsized":"2024-05-06T00:26:51.601148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:53:07.496607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.601391+0000","last_change":"2024-05-06T00:26:36.150377+0000","last_active":"2024-05-06T00:26:51.601391+0000","last_peered":"2024-05-06T00:26:51.601391+0000","last_clean":"2024-05-06T00:26:51.601391+0000","last_became_active":"2024-05-06T00:26:36.149496+0000","last_became_peered":"2024-05-06T00:26:36.149496+0000","last_unstale":"2024-05-06T00:26:51.601391+0000","last_undegraded":"2024-05-06T00:26:51.601391+0000","last_fullsized":"2024-05-06T00:26:51.601391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:54.410549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"26'21","reported_seq":53,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.488189+0000","last_change":"2024-05-06T00:26:45.712584+0000","last_active":"2024-05-06T00:26:52.488189+0000","last_peered":"2024-05-06T00:26:52.488189+0000","last_clean":"2024-05-06T00:26:52.488189+0000","last_became_active":"2024-05-06T00:26:43.688322+0000","last_became_peered":"2024-05-06T00:26:43.688322+0000","last_unstale":"2024-05-06T00:26:52.488189+0000","last_undegraded":"2024-05-06T00:26:52.488189+0000","last_fullsized":"2024-05-06T00:26:52.488189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:09:24.525375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":37,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":88,"num_read_kb":68,"num_write":32,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.602004+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:26:51.602004+0000","last_peered":"2024-05-06T00:26:51.602004+0000","last_clean":"2024-05-06T00:26:51.602004+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:26:51.602004+0000","last_undegraded":"2024-05-06T00:26:51.602004+0000","last_fullsized":"2024-05-06T00:26:51.602004+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.601257+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:26:51.601257+0000","last_peered":"2024-05-06T00:26:51.601257+0000","last_clean":"2024-05-06T00:26:51.601257+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:26:51.601257+0000","last_undegraded":"2024-05-06T00:26:51.601257+0000","last_fullsized":"2024-05-06T00:26:51.601257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:31:44.829951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.601686+0000","last_change":"2024-05-06T00:26:35.642448+0000","last_active":"2024-05-06T00:26:51.601686+0000","last_peered":"2024-05-06T00:26:51.601686+0000","last_clean":"2024-05-06T00:26:51.601686+0000","last_became_active":"2024-05-06T00:26:35.641678+0000","last_became_peered":"2024-05-06T00:26:35.641678+0000","last_unstale":"2024-05-06T00:26:51.601686+0000","last_undegraded":"2024-05-06T00:26:51.601686+0000","last_fullsized":"2024-05-06T00:26:51.601686+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:32:32.988296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"26'1","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.602379+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:26:51.602379+0000","last_peered":"2024-05-06T00:26:51.602379+0000","last_clean":"2024-05-06T00:26:51.602379+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:26:51.602379+0000","last_undegraded":"2024-05-06T00:26:51.602379+0000","last_fullsized":"2024-05-06T00:26:51.602379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:27:27.621545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.602192+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:26:51.602192+0000","last_peered":"2024-05-06T00:26:51.602192+0000","last_clean":"2024-05-06T00:26:51.602192+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:26:51.602192+0000","last_undegraded":"2024-05-06T00:26:51.602192+0000","last_fullsized":"2024-05-06T00:26:51.602192+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.685207+0000","last_change":"2024-05-06T00:26:45.733467+0000","last_active":"2024-05-06T00:26:51.685207+0000","last_peered":"2024-05-06T00:26:51.685207+0000","last_clean":"2024-05-06T00:26:51.685207+0000","last_became_active":"2024-05-06T00:26:43.703416+0000","last_became_peered":"2024-05-06T00:26:43.703416+0000","last_unstale":"2024-05-06T00:26:51.685207+0000","last_undegraded":"2024-05-06T00:26:51.685207+0000","last_fullsized":"2024-05-06T00:26:51.685207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:40:07.580059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"26'4","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.200056+0000","last_change":"2024-05-06T00:26:45.720211+0000","last_active":"2024-05-06T00:26:52.200056+0000","last_peered":"2024-05-06T00:26:52.200056+0000","last_clean":"2024-05-06T00:26:52.200056+0000","last_became_active":"2024-05-06T00:26:43.705235+0000","last_became_peered":"2024-05-06T00:26:43.705235+0000","last_unstale":"2024-05-06T00:26:52.200056+0000","last_undegraded":"2024-05-06T00:26:52.200056+0000","last_fullsized":"2024-05-06T00:26:52.200056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:14:37.757397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.684231+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:26:51.684231+0000","last_peered":"2024-05-06T00:26:51.684231+0000","last_clean":"2024-05-06T00:26:51.684231+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:26:51.684231+0000","last_undegraded":"2024-05-06T00:26:51.684231+0000","last_fullsized":"2024-05-06T00:26:51.684231+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"25'3","reported_seq":40,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.601896+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:26:51.601896+0000","last_peered":"2024-05-06T00:26:51.601896+0000","last_clean":"2024-05-06T00:26:51.601896+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:26:51.601896+0000","last_undegraded":"2024-05-06T00:26:51.601896+0000","last_fullsized":"2024-05-06T00:26:51.601896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:07:12.322473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":3,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.600752+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:26:51.600752+0000","last_peered":"2024-05-06T00:26:51.600752+0000","last_clean":"2024-05-06T00:26:51.600752+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:26:51.600752+0000","last_undegraded":"2024-05-06T00:26:51.600752+0000","last_fullsized":"2024-05-06T00:26:51.600752+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"26'12","reported_seq":125,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.484655+0000","last_change":"2024-05-06T00:26:45.712444+0000","last_active":"2024-05-06T00:26:52.484655+0000","last_peered":"2024-05-06T00:26:52.484655+0000","last_clean":"2024-05-06T00:26:52.484655+0000","last_became_active":"2024-05-06T00:26:43.708328+0000","last_became_peered":"2024-05-06T00:26:43.708328+0000","last_unstale":"2024-05-06T00:26:52.484655+0000","last_undegraded":"2024-05-06T00:26:52.484655+0000","last_fullsized":"2024-05-06T00:26:52.484655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:45:07.617464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":28869120,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":760,"num_write":14,"num_write_kb":269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.600677+0000","last_change":"2024-05-06T00:26:35.652664+0000","last_active":"2024-05-06T00:26:51.600677+0000","last_peered":"2024-05-06T00:26:51.600677+0000","last_clean":"2024-05-06T00:26:51.600677+0000","last_became_active":"2024-05-06T00:26:35.652017+0000","last_became_peered":"2024-05-06T00:26:35.652017+0000","last_unstale":"2024-05-06T00:26:51.600677+0000","last_undegraded":"2024-05-06T00:26:51.600677+0000","last_fullsized":"2024-05-06T00:26:51.600677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:34.935271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"26'8","reported_seq":54,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.436925+0000","last_change":"2024-05-06T00:26:45.725541+0000","last_active":"2024-05-06T00:26:52.436925+0000","last_peered":"2024-05-06T00:26:52.436925+0000","last_clean":"2024-05-06T00:26:52.436925+0000","last_became_active":"2024-05-06T00:26:43.696938+0000","last_became_peered":"2024-05-06T00:26:43.696938+0000","last_unstale":"2024-05-06T00:26:52.436925+0000","last_undegraded":"2024-05-06T00:26:52.436925+0000","last_fullsized":"2024-05-06T00:26:52.436925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:11:17.351889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":9184314,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":10,"num_write":7,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.684250+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:26:51.684250+0000","last_peered":"2024-05-06T00:26:51.684250+0000","last_clean":"2024-05-06T00:26:51.684250+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:26:51.684250+0000","last_undegraded":"2024-05-06T00:26:51.684250+0000","last_fullsized":"2024-05-06T00:26:51.684250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":126,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.600699+0000","last_change":"2024-05-06T00:26:30.239639+0000","last_active":"2024-05-06T00:26:51.600699+0000","last_peered":"2024-05-06T00:26:51.600699+0000","last_clean":"2024-05-06T00:26:51.600699+0000","last_became_active":"2024-05-06T00:26:30.237552+0000","last_became_peered":"2024-05-06T00:26:30.237552+0000","last_unstale":"2024-05-06T00:26:51.600699+0000","last_undegraded":"2024-05-06T00:26:51.600699+0000","last_fullsized":"2024-05-06T00:26:51.600699+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:44:57.011856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.600708+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:26:51.600708+0000","last_peered":"2024-05-06T00:26:51.600708+0000","last_clean":"2024-05-06T00:26:51.600708+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:26:51.600708+0000","last_undegraded":"2024-05-06T00:26:51.600708+0000","last_fullsized":"2024-05-06T00:26:51.600708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:42:51.206638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"26'3","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.449237+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:26:52.449237+0000","last_peered":"2024-05-06T00:26:52.449237+0000","last_clean":"2024-05-06T00:26:52.449237+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:26:52.449237+0000","last_undegraded":"2024-05-06T00:26:52.449237+0000","last_fullsized":"2024-05-06T00:26:52.449237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:09:42.790370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":33050624,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":2,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"26'9","reported_seq":40,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.601062+0000","last_change":"2024-05-06T00:26:45.710256+0000","last_active":"2024-05-06T00:26:51.601062+0000","last_peered":"2024-05-06T00:26:51.601062+0000","last_clean":"2024-05-06T00:26:51.601062+0000","last_became_active":"2024-05-06T00:26:43.710572+0000","last_became_peered":"2024-05-06T00:26:43.710572+0000","last_unstale":"2024-05-06T00:26:51.601062+0000","last_undegraded":"2024-05-06T00:26:51.601062+0000","last_fullsized":"2024-05-06T00:26:51.601062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:51:04.288169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":4366336,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":6,"num_write":7,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"24'5","reported_seq":40,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.602029+0000","last_change":"2024-05-06T00:26:45.725952+0000","last_active":"2024-05-06T00:26:51.602029+0000","last_peered":"2024-05-06T00:26:51.602029+0000","last_clean":"2024-05-06T00:26:51.602029+0000","last_became_active":"2024-05-06T00:26:43.686980+0000","last_became_peered":"2024-05-06T00:26:43.686980+0000","last_unstale":"2024-05-06T00:26:51.602029+0000","last_undegraded":"2024-05-06T00:26:51.602029+0000","last_fullsized":"2024-05-06T00:26:51.602029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:40:40.406896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.685131+0000","last_change":"2024-05-06T00:26:45.732742+0000","last_active":"2024-05-06T00:26:51.685131+0000","last_peered":"2024-05-06T00:26:51.685131+0000","last_clean":"2024-05-06T00:26:51.685131+0000","last_became_active":"2024-05-06T00:26:43.700115+0000","last_became_peered":"2024-05-06T00:26:43.700115+0000","last_unstale":"2024-05-06T00:26:51.685131+0000","last_undegraded":"2024-05-06T00:26:51.685131+0000","last_fullsized":"2024-05-06T00:26:51.685131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:47:28.066501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"23'2","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.601466+0000","last_change":"2024-05-06T00:26:45.711599+0000","last_active":"2024-05-06T00:26:51.601466+0000","last_peered":"2024-05-06T00:26:51.601466+0000","last_clean":"2024-05-06T00:26:51.601466+0000","last_became_active":"2024-05-06T00:26:43.692092+0000","last_became_peered":"2024-05-06T00:26:43.692092+0000","last_unstale":"2024-05-06T00:26:51.601466+0000","last_undegraded":"2024-05-06T00:26:51.601466+0000","last_fullsized":"2024-05-06T00:26:51.601466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:04:11.872310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"26'40","reported_seq":526,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.256102+0000","last_change":"2024-05-06T00:26:45.734276+0000","last_active":"2024-05-06T00:26:52.256102+0000","last_peered":"2024-05-06T00:26:52.256102+0000","last_clean":"2024-05-06T00:26:52.256102+0000","last_became_active":"2024-05-06T00:26:43.711491+0000","last_became_peered":"2024-05-06T00:26:43.711491+0000","last_unstale":"2024-05-06T00:26:52.256102+0000","last_undegraded":"2024-05-06T00:26:52.256102+0000","last_fullsized":"2024-05-06T00:26:52.256102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:42:37.592320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":9859072,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":941,"num_read_kb":1995,"num_write":20,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.685221+0000","last_change":"2024-05-06T00:26:45.734362+0000","last_active":"2024-05-06T00:26:51.685221+0000","last_peered":"2024-05-06T00:26:51.685221+0000","last_clean":"2024-05-06T00:26:51.685221+0000","last_became_active":"2024-05-06T00:26:43.703972+0000","last_became_peered":"2024-05-06T00:26:43.703972+0000","last_unstale":"2024-05-06T00:26:51.685221+0000","last_undegraded":"2024-05-06T00:26:51.685221+0000","last_fullsized":"2024-05-06T00:26:51.685221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:11:44.243380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.684488+0000","last_change":"2024-05-06T00:26:45.732061+0000","last_active":"2024-05-06T00:26:51.684488+0000","last_peered":"2024-05-06T00:26:51.684488+0000","last_clean":"2024-05-06T00:26:51.684488+0000","last_became_active":"2024-05-06T00:26:43.699883+0000","last_became_peered":"2024-05-06T00:26:43.699883+0000","last_unstale":"2024-05-06T00:26:51.684488+0000","last_undegraded":"2024-05-06T00:26:51.684488+0000","last_fullsized":"2024-05-06T00:26:51.684488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:08:33.909345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.600848+0000","last_change":"2024-05-06T00:26:45.710994+0000","last_active":"2024-05-06T00:26:51.600848+0000","last_peered":"2024-05-06T00:26:51.600848+0000","last_clean":"2024-05-06T00:26:51.600848+0000","last_became_active":"2024-05-06T00:26:43.710745+0000","last_became_peered":"2024-05-06T00:26:43.710745+0000","last_unstale":"2024-05-06T00:26:51.600848+0000","last_undegraded":"2024-05-06T00:26:51.600848+0000","last_fullsized":"2024-05-06T00:26:51.600848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:45:44.512026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"24'7","reported_seq":82,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:51.681884+0000","last_change":"2024-05-06T00:26:45.713363+0000","last_active":"2024-05-06T00:26:51.681884+0000","last_peered":"2024-05-06T00:26:51.681884+0000","last_clean":"2024-05-06T00:26:51.681884+0000","last_became_active":"2024-05-06T00:26:43.692971+0000","last_became_peered":"2024-05-06T00:26:43.692971+0000","last_unstale":"2024-05-06T00:26:51.681884+0000","last_undegraded":"2024-05-06T00:26:51.681884+0000","last_fullsized":"2024-05-06T00:26:51.681884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:06:20.404224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":55,"num_write":7,"num_write_kb":156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"26'28","reported_seq":343,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-06T00:26:52.491677+0000","last_change":"2024-05-06T00:26:45.711743+0000","last_active":"2024-05-06T00:26:52.491677+0000","last_peered":"2024-05-06T00:26:52.491677+0000","last_clean":"2024-05-06T00:26:52.491677+0000","last_became_active":"2024-05-06T00:26:43.700214+0000","last_became_peered":"2024-05-06T00:26:43.700214+0000","last_unstale":"2024-05-06T00:26:52.491677+0000","last_undegraded":"2024-05-06T00:26:52.491677+0000","last_fullsized":"2024-05-06T00:26:52.491677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:46:23.716846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":16777216,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":499,"num_read_kb":820,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":183427333,"num_objects":40,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":6,"num_read":4033,"num_read_kb":8297,"num_write":350,"num_write_kb":2063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1609226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":347,"ondisk_log_size":347,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640266,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":438272,"data_stored":243644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640264,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31752,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340088,"statfs":{"total":96636764160,"available":96604250112,"internally_reserved":0,"allocated":585728,"data_stored":391566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":909312,"data_stored":725934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":98664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640266,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":425984,"data_stored":221658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1073152,"data_stored":866848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":318120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":324738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":79888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":41014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":634880,"data_stored":590390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":255076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:26:56.712 INFO:tasks.thrashosds.thrasher:Setting 3.6 to [2, 5, 1, 4] 2024-05-06T00:26:56.712 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.6', '2', '5', '1', '4'] 2024-05-06T00:26:56.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.6 2 5 1 4 2024-05-06T00:26:56.726 INFO:teuthology.orchestra.run.smithi186.stdout:552 trunc from 0xb4d0200 to 0x11eea00 2024-05-06T00:26:56.740 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:56.740 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:56.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:26:56.761 INFO:teuthology.orchestra.run.smithi186.stdout:553 punch from 0x11cb000 to 0x11d0000, (0x5000 bytes) 2024-05-06T00:26:56.763 INFO:teuthology.orchestra.run.smithi186.stdout:554 punch from 0xc76c00 to 0xc7ac00, (0x4000 bytes) 2024-05-06T00:26:56.783 INFO:teuthology.orchestra.run.smithi186.stdout:555 write 0x4622200 thru 0x462ea57 (0xc858 bytes) 2024-05-06T00:26:56.801 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:56.801+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:56.802 INFO:teuthology.orchestra.run.smithi186.stdout:557 write 0x5ea0200 thru 0x5eaf9f4 (0xf7f5 bytes) 2024-05-06T00:26:56.804 INFO:teuthology.orchestra.run.smithi186.stdout:558 write 0x28ee000 thru 0x28f3665 (0x5666 bytes) 2024-05-06T00:26:56.810 INFO:teuthology.orchestra.run.smithi186.stdout:559 trunc from 0x5eafa00 to 0x8b2a400 2024-05-06T00:26:56.826 INFO:teuthology.orchestra.run.smithi186.stdout:560 punch from 0x1d50c00 to 0x1d5ac00, (0xa000 bytes) 2024-05-06T00:26:56.828 INFO:teuthology.orchestra.run.smithi186.stdout:561 read 0x65bc800 thru 0x65bd466 (0xc67 bytes) 2024-05-06T00:26:56.838 INFO:teuthology.orchestra.run.smithi186.stdout:562 trunc from 0x8b2a400 to 0x2ace600 2024-05-06T00:26:56.862 INFO:teuthology.orchestra.run.smithi186.stdout:563 read 0x18b8800 thru 0x18c4f41 (0xc742 bytes) 2024-05-06T00:26:56.863 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:56.861+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:56.864 INFO:teuthology.orchestra.run.smithi186.stdout:565 read 0x1223a00 thru 0x1223cb0 (0x2b1 bytes) 2024-05-06T00:26:56.865 INFO:teuthology.orchestra.run.smithi186.stdout:566 punch from 0x26a5200 to 0x26a8800, (0x3600 bytes) 2024-05-06T00:26:56.892 INFO:teuthology.orchestra.run.smithi186.stdout:567 write 0xc21bc00 thru 0xc225dd1 (0xa1d2 bytes) 2024-05-06T00:26:56.893 INFO:teuthology.orchestra.run.smithi186.stdout:570 read 0x74fea00 thru 0x75068d5 (0x7ed6 bytes) 2024-05-06T00:26:56.896 INFO:teuthology.orchestra.run.smithi186.stdout:572 trunc from 0xc225e00 to 0xdb7a000 2024-05-06T00:26:56.912 INFO:teuthology.orchestra.run.smithi186.stdout:573 punch from 0xd6eb600 to 0xd6fb200, (0xfc00 bytes) 2024-05-06T00:26:56.913 INFO:teuthology.orchestra.run.smithi186.stdout:574 write 0xbc92c00 thru 0xbc9408f (0x1490 bytes) 2024-05-06T00:26:56.913 INFO:teuthology.orchestra.run.smithi186.stdout:576 read 0x6a8c800 thru 0x6a93f9d (0x779e bytes) 2024-05-06T00:26:56.913 INFO:teuthology.orchestra.run.smithi186.stdout:578 read 0x2171c00 thru 0x2175b02 (0x3f03 bytes) 2024-05-06T00:26:56.927 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:56.927 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:56.934 INFO:teuthology.orchestra.run.smithi186.stdout:579 trunc from 0xdb7a000 to 0x9e7c00 2024-05-06T00:26:56.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:26:56.968 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:56.961+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:56.974 INFO:teuthology.orchestra.run.smithi186.stdout:580 read 0x3ab000 thru 0x3b1e3d (0x6e3e bytes) 2024-05-06T00:26:56.976 INFO:teuthology.orchestra.run.smithi186.stdout:583 read 0x552600 thru 0x55e74d (0xc14e bytes) 2024-05-06T00:26:57.004 INFO:teuthology.orchestra.run.smithi186.stdout:585 write 0x759d000 thru 0x75a3ef3 (0x6ef4 bytes) 2024-05-06T00:26:57.005 INFO:teuthology.orchestra.run.smithi186.stdout:589 write 0x2316c00 thru 0x231b246 (0x4647 bytes) 2024-05-06T00:26:57.005 INFO:teuthology.orchestra.run.smithi186.stdout:590 write 0x5e27000 thru 0x5e34313 (0xd314 bytes) 2024-05-06T00:26:57.005 INFO:teuthology.orchestra.run.smithi186.stdout:591 read 0x4b70c00 thru 0x4b77445 (0x6846 bytes) 2024-05-06T00:26:57.049 INFO:teuthology.orchestra.run.smithi186.stdout:595 write 0xd4e0a00 thru 0xd4e7d53 (0x7354 bytes) 2024-05-06T00:26:57.052 INFO:teuthology.orchestra.run.smithi186.stdout:596 punch from 0xd6600 to 0xe0400, (0x9e00 bytes) 2024-05-06T00:26:57.053 INFO:teuthology.orchestra.run.smithi186.stdout:597 read 0x3af1c00 thru 0x3af4dfa (0x31fb bytes) 2024-05-06T00:26:57.053 INFO:teuthology.orchestra.run.smithi186.stdout:600 write 0x4a77a00 thru 0x4a7d0f6 (0x56f7 bytes) 2024-05-06T00:26:57.053 INFO:teuthology.orchestra.run.smithi186.stdout:601 write 0x1118800 thru 0x111ab35 (0x2336 bytes) 2024-05-06T00:26:57.055 INFO:teuthology.orchestra.run.smithi186.stdout:603 write 0x6f3400 thru 0x6f9cd1 (0x68d2 bytes) 2024-05-06T00:26:57.058 INFO:teuthology.orchestra.run.smithi186.stdout:604 read 0x86e9200 thru 0x86f1829 (0x862a bytes) 2024-05-06T00:26:57.067 INFO:teuthology.orchestra.run.smithi186.stdout:605 trunc from 0xd4e7e00 to 0xa49de00 2024-05-06T00:26:57.067 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:57.065+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:57.115 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:57.116 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:57.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:26:57.131 INFO:teuthology.orchestra.run.smithi186.stdout:610 write 0xe8ac400 thru 0xe8ba5f1 (0xe1f2 bytes) 2024-05-06T00:26:57.149 INFO:teuthology.orchestra.run.smithi186.stdout:611 trunc from 0xe8ba600 to 0x4880e00 2024-05-06T00:26:57.173 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:57.173+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:57.231 INFO:teuthology.orchestra.run.smithi186.stdout:614 write 0xae25a00 thru 0xae2e86d (0x8e6e bytes) 2024-05-06T00:26:57.233 INFO:teuthology.orchestra.run.smithi186.stdout:616 punch from 0xa5c1c00 to 0xa5c9c00, (0x8000 bytes) 2024-05-06T00:26:57.233 INFO:teuthology.orchestra.run.smithi186.stdout:617 read 0xbfd400 thru 0xc00654 (0x3255 bytes) 2024-05-06T00:26:57.265 INFO:teuthology.orchestra.run.smithi186.stdout:618 write 0xe50ba00 thru 0xe50c576 (0xb77 bytes) 2024-05-06T00:26:57.267 INFO:teuthology.orchestra.run.smithi186.stdout:621 punch from 0xaa2fc00 to 0xaa3d200, (0xd600 bytes) 2024-05-06T00:26:57.268 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:57.269+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:57.299 INFO:teuthology.orchestra.run.smithi186.stdout:625 write 0xe69ca00 thru 0xe6a737d (0xa97e bytes) 2024-05-06T00:26:57.301 INFO:teuthology.orchestra.run.smithi186.stdout:626 write 0xc990000 thru 0xc99372b (0x372c bytes) 2024-05-06T00:26:57.301 INFO:teuthology.orchestra.run.smithi186.stdout:627 punch from 0x13f3800 to 0x13f8a00, (0x5200 bytes) 2024-05-06T00:26:57.302 INFO:teuthology.orchestra.run.smithi186.stdout:628 write 0x58a4400 thru 0x58a8ef5 (0x4af6 bytes) 2024-05-06T00:26:57.302 INFO:teuthology.orchestra.run.smithi186.stdout:630 punch from 0xa027a00 to 0xa030e00, (0x9400 bytes) 2024-05-06T00:26:57.302 INFO:teuthology.orchestra.run.smithi186.stdout:631 clone 4 order 24 su 1048576 sc 9 2024-05-06T00:26:57.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:26:57.396 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:57.397+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:57.470 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:57.469+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:57.472 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:57.472 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:57.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:26:57.572 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:57.573+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:57.645 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:57.646 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:57.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:26:57.704 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:57.705+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:57.772 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:57.769+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:57.786 INFO:teuthology.orchestra.run.smithi078.stderr:set 3.6 pg_upmap_items mapping to [2->5,1->4] 2024-05-06T00:26:57.831 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:57.831 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:57.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:26:57.858 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:26:57.910 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:57.909+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:57.969 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:57.969 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:57.973 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:57.973+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:57.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:26:58.074 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:26:58.073+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:26:58.100 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:58.100 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:58.115 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:26:58.210 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:58.209+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:58.277 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:58.277+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:58.284 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:58.284 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:58.299 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:26:58.377 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:58.377+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:58.463 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:58.463 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:58.476 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:26:58.478 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:58.477+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:58.486 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:26:58.578 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:58.577+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:58.600 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:26:58.600 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:26:58.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:26:58.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:26:58.659 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:26:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:26:58.676 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:26:58.679 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:58.681+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:58.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:26:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:26:58.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:26:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:26:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:58.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:58.730 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:58.730 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:58.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:26:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:26:58.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:26:58.755 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:26:58.755 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:26:58.780 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:26:58.781+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:26:58.804 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:26:58.881 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:58.877+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:58.883 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:26:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:26:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:26:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:228 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 397312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:26:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:26:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.2839100949999995, 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072743440000000003, 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 228 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597016+0000", 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.443e-06 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597023+0000", 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8340000000000001e-06 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597044+0000", 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0876000000000001e-05 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597055+0000", 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1314e-05 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597176+0000", 2024-05-06T00:26:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120565 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597192+0000", 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6337e-05 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603743+0000", 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065506820000000004 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603976+0000", 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000233355 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604289+0000", 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031293800000000002 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:229 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 528384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.283752196, 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075584060000000002, 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 229 2024-05-06T00:26:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597175+0000", 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8929999999999999e-06 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597184+0000", 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2040000000000003e-06 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597200+0000", 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5642999999999999e-05 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597209+0000", 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4329999999999993e-06 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597301+0000", 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1588999999999993e-05 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597309+0000", 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604360+0000", 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070513030000000001 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604516+0000", 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015564999999999999 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604731+0000", 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000215267 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:91 3.6 e28/18 3:645d64d7:::rbd_header.110cc8beeb04:head v 28'129, mlcod=28'126)", 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.608485+0000", 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2724400390000001, 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054914009999999999, 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608485+0000", 2024-05-06T00:26:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608485+0000", 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608486+0000", 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0759999999999999e-06 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608497+0000", 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0801999999999999e-05 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608510+0000", 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.331e-05 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608519+0000", 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6090000000000001e-06 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608598+0000", 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9648999999999996e-05 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.608649+0000", 2024-05-06T00:26:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0407e-05 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.613919+0000", 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052703230000000004 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.613976+0000", 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7225000000000002e-05 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:161 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.683279+0000", 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.197646057, 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054288820000000003, 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 161 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683279+0000", 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683279+0000", 2024-05-06T00:26:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683281+0000", 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.486e-06 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683293+0000", 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1404e-05 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683324+0000", 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1409999999999999e-05 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683339+0000", 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4513e-05 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683426+0000", 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7640999999999997e-05 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683526+0000", 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010002599999999999 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683989+0000", 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00046260000000000002 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687768+0000", 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003778862 2024-05-06T00:26:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688605+0000", 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00083695299999999998 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688655+0000", 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0179999999999997e-05 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688708+0000", 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2806999999999998e-05 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:163 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1972975670000001, 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060279510000000001, 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 163 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:26:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683629+0000", 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.28e-06 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683636+0000", 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2729999999999997e-06 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683662+0000", 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5754000000000001e-05 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683673+0000", 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0890000000000001e-05 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.686748+0000", 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030753180000000001 2024-05-06T00:26:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.686848+0000", 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9697000000000004e-05 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687256+0000", 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040807300000000001 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688723+0000", 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014670740000000001 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689566+0000", 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00084270500000000002 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689616+0000", 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0358999999999997e-05 2024-05-06T00:26:58.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689655+0000", 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9527999999999998e-05 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:165 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1967684539999999, 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010266747, 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 165 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684158+0000", 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6929999999999999e-06 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684170+0000", 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1933000000000001e-05 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684198+0000", 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8266999999999999e-05 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684221+0000", 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2736999999999999e-05 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687824+0000", 2024-05-06T00:26:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003602863 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687915+0000", 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0804999999999995e-05 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688264+0000", 2024-05-06T00:26:58.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034917100000000001 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693979+0000", 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057148959999999997 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694148+0000", 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168951 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694315+0000", 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000167016 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694423+0000", 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108415 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:173 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1912579700000001, 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058174089999999999, 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 173 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689668+0000", 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2669999999999999e-06 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689676+0000", 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8569999999999995e-06 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689696+0000", 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0006000000000001e-05 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689711+0000", 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-06T00:26:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.691939+0000", 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002227559 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.692171+0000", 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000232475 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693111+0000", 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000939795 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694929+0000", 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001817281 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695267+0000", 2024-05-06T00:26:58.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033894600000000002 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695385+0000", 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117683 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695484+0000", 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9343000000000001e-05 2024-05-06T00:26:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:215 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.718165+0000", 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1627603820000001, 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054499259999999999, 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 215 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718165+0000", 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718165+0000", 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718165+0000", 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6199999999999997e-07 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718170+0000", 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3780000000000003e-06 2024-05-06T00:26:58.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718178+0000", 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.807e-06 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718183+0000", 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9030000000000001e-06 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718800+0000", 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000616571 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718857+0000", 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6648999999999999e-05 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.719048+0000", 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019183500000000001 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.720699+0000", 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001650119 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723517+0000", 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002818442 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723575+0000", 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8301000000000003e-05 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723615+0000", 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9258999999999998e-05 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:216 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.718348+0000", 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1625774469999999, 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005402029, 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 216 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718348+0000", 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718348+0000", 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718348+0000", 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.16e-07 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718353+0000", 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5539999999999999e-06 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718362+0000", 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0049999999999993e-06 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718370+0000", 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1140000000000006e-06 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.719307+0000", 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00093682699999999999 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.719351+0000", 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4107000000000003e-05 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.719651+0000", 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029977500000000001 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723309+0000", 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003657759 2024-05-06T00:26:58.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723659+0000", 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035054800000000001 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723707+0000", 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7587999999999999e-05 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723750+0000", 2024-05-06T00:26:58.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2935999999999997e-05 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:219 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.718604+0000", 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1623206619999999, 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055381600000000003, 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 219 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718604+0000", 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718604+0000", 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718605+0000", 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4300000000000003e-07 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718608+0000", 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4680000000000001e-06 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718626+0000", 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7252999999999999e-05 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.718631+0000", 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7679999999999997e-06 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.720838+0000", 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022063389999999999 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.721063+0000", 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022549899999999999 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.721885+0000", 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00082158299999999997 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723762+0000", 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018767429999999999 2024-05-06T00:26:58.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724044+0000", 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028263499999999999 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724110+0000", 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5297999999999997e-05 2024-05-06T00:26:58.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724143+0000", 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3031000000000003e-05 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:278 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.947 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.758895+0000", 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1220304169999999, 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054978930000000002, 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 278 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.948 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.758895+0000", 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.758895+0000", 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.758897+0000", 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1730000000000002e-06 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.758908+0000", 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0847e-05 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.758937+0000", 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9436000000000001e-05 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.758952+0000", 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759051+0000", 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9032999999999999e-05 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759182+0000", 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013077499999999999 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759828+0000", 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064658800000000002 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763553+0000", 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003725164 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763753+0000", 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199552 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.764291+0000", 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00053792399999999995 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.764393+0000", 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101771 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:283 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.759079+0000", 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1218456890000001, 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055198010000000004, 2024-05-06T00:26:58.954 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 283 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759079+0000", 2024-05-06T00:26:58.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759079+0000", 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759080+0000", 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3699999999999999e-07 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759086+0000", 2024-05-06T00:26:58.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2329999999999997e-06 2024-05-06T00:26:58.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759099+0000", 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3927000000000001e-05 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759108+0000", 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.106e-06 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.760840+0000", 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001732075 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.761051+0000", 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021123299999999999 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.761806+0000", 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00075511399999999998 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763723+0000", 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019169020000000001 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.764418+0000", 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000694898 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.764517+0000", 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8770000000000005e-05 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.764599+0000", 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2606000000000002e-05 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:298 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.121433803, 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089449070000000002, 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 298 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759493+0000", 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8029999999999999e-06 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759503+0000", 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4769999999999992e-06 2024-05-06T00:26:58.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759525+0000", 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2067999999999999e-05 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759540+0000", 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5296999999999999e-05 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.762625+0000", 2024-05-06T00:26:58.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030855399999999999 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.762723+0000", 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7750000000000004e-05 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763171+0000", 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044755199999999999 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766326+0000", 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031554080000000002 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768328+0000", 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020015530000000001 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768387+0000", 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9283000000000001e-05 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768436+0000", 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9175999999999998e-05 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:300 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.114511501, 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077869669999999997, 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.967 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 300 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766415+0000", 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.536e-06 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766423+0000", 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2849999999999995e-06 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766435+0000", 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2121e-05 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766443+0000", 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9140000000000008e-06 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766490+0000", 2024-05-06T00:26:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6517999999999997e-05 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766495+0000", 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769158+0000", 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026631390000000001 2024-05-06T00:26:58.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769268+0000", 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010981599999999999 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769603+0000", 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033493500000000002 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.773467+0000", 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038639920000000001 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774044+0000", 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00057753000000000004 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774139+0000", 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5315999999999993e-05 2024-05-06T00:26:58.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774201+0000", 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1054000000000002e-05 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:324 3.19 3:99a6ca97:::rbd_data.110cc8beeb04.0000000000000013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:58.974 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.0899160659999998, 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062413850000000003, 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 324 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.975 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791010+0000", 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0449999999999999e-06 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791015+0000", 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4689999999999996e-06 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791030+0000", 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4892000000000001e-05 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791038+0000", 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8460000000000008e-06 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791106+0000", 2024-05-06T00:26:58.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7994e-05 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791190+0000", 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3337000000000005e-05 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791293+0000", 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103757 2024-05-06T00:26:58.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795405+0000", 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041115479999999996 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797183+0000", 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001778097 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797213+0000", 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.004e-05 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797250+0000", 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7360000000000001e-05 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:328 3.19 3:9fbbd60a:::rbd_data.110cc8beeb04.000000000000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.0897677539999999, 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063090830000000001, 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:58.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:58.983 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 328 2024-05-06T00:26:58.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.983 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:26:58.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791158+0000", 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7000000000000003e-07 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791165+0000", 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3220000000000004e-06 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791178+0000", 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2078e-05 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791184+0000", 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791609+0000", 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042469600000000001 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791782+0000", 2024-05-06T00:26:58.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017316400000000001 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.792020+0000", 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023781599999999999 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795867+0000", 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038475010000000001 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797389+0000", 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015214339999999999 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797418+0000", 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8977999999999999e-05 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797466+0000", 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8489999999999998e-05 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1015 3.8 e29/18 3:1391aa96:::rbd_data.110cc8beeb04.0000000000000025:head v 29'6, mlcod=29'4)", 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.626862649, 2024-05-06T00:26:58.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075649460000000003, 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254063+0000", 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4099999999999997e-07 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254072+0000", 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6370000000000005e-06 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254084+0000", 2024-05-06T00:26:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1725999999999999e-05 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254091+0000", 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.379e-06 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254155+0000", 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3906999999999996e-05 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254177+0000", 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1526e-05 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261558+0000", 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073814409999999999 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261627+0000", 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9388999999999996e-05 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1017 3.7 e29/18 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head v 29'11, mlcod=29'9)", 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.626817661, 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075152739999999997, 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254108+0000", 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254112+0000", 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254119+0000", 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7800000000000003e-06 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254124+0000", 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8509999999999998e-06 2024-05-06T00:26:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254189+0000", 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4757000000000004e-05 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254209+0000", 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0307000000000001e-05 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261566+0000", 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007356596 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261623+0000", 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7111000000000003e-05 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1022 3.8 e29/18 3:17fe3906:::rbd_data.110cc8beeb04.0000000000000036:head v 29'7, mlcod=29'4)", 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.6266555450000002, 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073889450000000001, 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:26:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:26:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254270+0000", 2024-05-06T00:26:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-05-06T00:26:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254275+0000", 2024-05-06T00:26:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6120000000000001e-06 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254283+0000", 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2199999999999992e-06 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254289+0000", 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9880000000000001e-06 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254360+0000", 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1062000000000005e-05 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254379+0000", 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9194e-05 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261632+0000", 2024-05-06T00:26:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00725255 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261658+0000", 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6857000000000001e-05 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1018 3.7 e29/18 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head v 29'12, mlcod=29'9)", 2024-05-06T00:26:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.6266037180000001, 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073213799999999997, 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254322+0000", 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5400000000000002e-07 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254326+0000", 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5979999999999999e-06 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254334+0000", 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1489999999999997e-06 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254339+0000", 2024-05-06T00:26:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1399999999999999e-06 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254387+0000", 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8102000000000001e-05 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254403+0000", 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5982e-05 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261626+0000", 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072235140000000003 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261643+0000", 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6441000000000001e-05 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.011 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.011 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:59.013 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:26:59.013 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:26:59.009+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:26:59.113 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:59.113+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:59.182 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:59.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:59.183 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:26:59.183 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:59.198 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:26:59.242 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:59.241+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:59.315 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:59.313+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:59.315 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:26:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:26:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:26:59.315 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:59.330 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:26:59.418 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:59.417+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:223 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3702784~58880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7746008089999998, 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005189656, 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 223 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723867+0000", 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8569999999999999e-06 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724065+0000", 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019875599999999999 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724102+0000", 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6362999999999997e-05 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724122+0000", 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0519000000000001e-05 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724220+0000", 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7466000000000005e-05 2024-05-06T00:26:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724367+0000", 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014768100000000001 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724516+0000", 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014873800000000001 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728904+0000", 2024-05-06T00:26:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043882369999999997 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728924+0000", 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9959999999999999e-05 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728992+0000", 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7519000000000002e-05 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729053+0000", 2024-05-06T00:26:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.156e-05 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:224 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2300416~15872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7742855940000002, 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062082700000000001, 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 224 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724180+0000", 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.471e-06 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724206+0000", 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729e-05 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724221+0000", 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5114e-05 2024-05-06T00:26:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724230+0000", 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2369999999999998e-06 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725101+0000", 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087058999999999997 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725503+0000", 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000402474 2024-05-06T00:26:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725876+0000", 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037291499999999998 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729081+0000", 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032042339999999998 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730250+0000", 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001168955 2024-05-06T00:26:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730326+0000", 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6148000000000003e-05 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730387+0000", 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1402999999999995e-05 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:225 3.7 3:e39c62f0:::rbd_data.110cc8beeb04.0000000000000012:head [set-alloc-hint object_size 4194304 write_size 4194304,write 905216~12288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7742152660000001, 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062774959999999996, 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 225 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724250+0000", 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.71e-07 2024-05-06T00:26:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724265+0000", 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.502e-05 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724275+0000", 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7569999999999999e-06 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724282+0000", 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1860000000000003e-06 2024-05-06T00:26:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.727270+0000", 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029884489999999998 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.727639+0000", 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036815399999999999 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728041+0000", 2024-05-06T00:26:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000402013 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729289+0000", 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012479399999999999 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730444+0000", 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001155498 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730493+0000", 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8727999999999998e-05 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730527+0000", 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3880000000000001e-05 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:298 3.4 e28/18 3:26d09f8b:::rbd_object_map.110cc8beeb04:head v 28'28, mlcod=28'24)", 2024-05-06T00:26:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7351687330000001, 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048531540000000001, 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:26:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763298+0000", 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9630000000000001e-06 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763310+0000", 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2782999999999999e-05 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763333+0000", 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2025e-05 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763351+0000", 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8882999999999999e-05 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763872+0000", 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00052017299999999997 2024-05-06T00:26:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763924+0000", 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2473999999999999e-05 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.767937+0000", 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040133399999999998 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768149+0000", 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021151300000000001 2024-05-06T00:26:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:322 3.7 3:e39c62f0:::rbd_data.110cc8beeb04.0000000000000012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.707494563, 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050787829999999999, 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 322 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790972+0000", 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.731e-06 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790977+0000", 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.933e-06 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791004+0000", 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6974999999999998e-05 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791012+0000", 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6070000000000008e-06 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791099+0000", 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6571999999999997e-05 2024-05-06T00:26:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791197+0000", 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8072000000000006e-05 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791302+0000", 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105414 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.792824+0000", 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001521912 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795944+0000", 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031202629999999999 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796002+0000", 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7547999999999999e-05 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796049+0000", 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6755999999999998e-05 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:325 3.9 e29/18 3:90865cbe:::rbd_data.110cc8beeb04.0000000000000015:head v 29'19, mlcod=28'17)", 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.794576+0000", 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7038880289999998, 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003977721, 2024-05-06T00:26:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.794576+0000", 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.794576+0000", 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.794577+0000", 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.794583+0000", 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.592e-06 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.794593+0000", 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0216e-05 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.794605+0000", 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1875000000000001e-05 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.794652+0000", 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6999999999999997e-05 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.794666+0000", 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3935e-05 2024-05-06T00:26:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.798453+0000", 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037876810000000002 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.798554+0000", 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010077300000000001 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:348 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7021332610000002, 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048444150000000004, 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 348 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796332+0000", 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0240000000000001e-06 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796336+0000", 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2710000000000003e-06 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796348+0000", 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1843e-05 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796355+0000", 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9129999999999997e-06 2024-05-06T00:26:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796409+0000", 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3698e-05 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2955999999999997e-05 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796581+0000", 2024-05-06T00:26:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9319000000000002e-05 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.798666+0000", 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020843089999999999 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801063+0000", 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002397518 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801141+0000", 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7948000000000007e-05 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801176+0000", 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4616000000000002e-05 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:351 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7019826849999999, 2024-05-06T00:26:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049431070000000004, 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 351 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.37e-07 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796485+0000", 2024-05-06T00:26:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.582e-06 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796495+0000", 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6350000000000001e-06 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796499+0000", 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3100000000000002e-06 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796827+0000", 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032834500000000001 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796904+0000", 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6808000000000001e-05 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797004+0000", 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9998e-05 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801185+0000", 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041808089999999997 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801284+0000", 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9228999999999995e-05 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801363+0000", 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9214000000000004e-05 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801425+0000", 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1639999999999999e-05 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:343 3.9 e29/18 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head v 29'20, mlcod=28'17)", 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.796995+0000", 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.701469157, 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041966920000000001, 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796995+0000", 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796995+0000", 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796996+0000", 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1699999999999997e-07 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797029+0000", 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3250000000000002e-05 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797038+0000", 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3769999999999996e-06 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797044+0000", 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4330000000000001e-06 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797105+0000", 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0775e-05 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797134+0000", 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8880000000000001e-05 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801116+0000", 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039818290000000001 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801192+0000", 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6330999999999998e-05 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:833 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [truncate 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.3642028549999998, 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047565890000000003, 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 833 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2099999999999996e-07 2024-05-06T00:26:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134265+0000", 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5909999999999998e-06 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134273+0000", 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5599999999999994e-06 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134279+0000", 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2580000000000003e-06 2024-05-06T00:26:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134801+0000", 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00052267500000000001 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134865+0000", 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3507e-05 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134939+0000", 2024-05-06T00:26:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4564e-05 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.136596+0000", 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016564749999999999 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.138912+0000", 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002316214 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.138974+0000", 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1941999999999997e-05 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.139018+0000", 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4181999999999998e-05 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1015 3.8 3:1391aa96:::rbd_data.110cc8beeb04.0000000000000025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:26:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2449833450000001, 2024-05-06T00:26:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084885580000000002, 2024-05-06T00:26:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253483+0000", 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0090000000000002e-06 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253489+0000", 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2639999999999997e-06 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253510+0000", 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0462000000000002e-05 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253519+0000", 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5550000000000008e-06 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253594+0000", 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4345999999999996e-05 2024-05-06T00:26:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253675+0000", 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0860999999999997e-05 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253779+0000", 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104353 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.256496+0000", 2024-05-06T00:26:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002716703 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261853+0000", 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053571549999999997 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261907+0000", 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4381e-05 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261970+0000", 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2469000000000001e-05 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1017 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2449201470000002, 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084085759999999992, 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1017 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:26:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253545+0000", 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6000000000000003e-07 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253549+0000", 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6710000000000001e-06 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253557+0000", 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253563+0000", 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9440000000000002e-06 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253683+0000", 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120147 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253759+0000", 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5817999999999998e-05 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253858+0000", 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9053000000000002e-05 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259253+0000", 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005395047 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261834+0000", 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025814929999999998 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261900+0000", 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5738e-05 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261953+0000", 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2784000000000001e-05 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1018 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2448829379999999, 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085355099999999996, 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4199999999999996e-07 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253587+0000", 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7119999999999999e-06 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253595+0000", 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7980000000000006e-06 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253601+0000", 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254070+0000", 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00046893599999999998 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254144+0000", 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4943000000000004e-05 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254244+0000", 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9091000000000004e-05 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259286+0000", 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050428850000000004 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262019+0000", 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027324430000000002 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262072+0000", 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2994000000000001e-05 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262117+0000", 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5155999999999999e-05 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1022 3.8 3:17fe3906:::rbd_data.110cc8beeb04.0000000000000036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2448553310000001, 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085045799999999994, 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1022 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253610+0000", 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.51e-07 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253612+0000", 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2979999999999999e-06 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253618+0000", 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.426e-06 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253623+0000", 2024-05-06T00:26:59.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7260000000000002e-06 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254047+0000", 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042410499999999998 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254100+0000", 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2592000000000001e-05 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:26:59.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254190+0000", 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0001999999999997e-05 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259272+0000", 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050825840000000002 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262019+0000", 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00274654 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262071+0000", 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2091e-05 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262114+0000", 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2664999999999999e-05 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1042 3.1e 3:7e323eed:::rbd_data.110cc8beeb04.0000000000000002:head [stat,truncate 3670016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.580 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.233360936, 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059170580000000002, 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1042 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:26:59.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265105+0000", 2024-05-06T00:26:59.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.857e-06 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265112+0000", 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7769999999999997e-06 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265131+0000", 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8805e-05 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265144+0000", 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3474e-05 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265260+0000", 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000115785 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265343+0000", 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3010000000000007e-05 2024-05-06T00:26:59.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265472+0000", 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128625 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.266670+0000", 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001198382 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.270924+0000", 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042537260000000002 2024-05-06T00:26:59.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.270974+0000", 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0484e-05 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271021+0000", 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6133e-05 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1045 3.1e 3:7ff04bbe:::rbd_data.110cc8beeb04.0000000000000005:head [stat,truncate 3538944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.23329171, 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005939003, 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1045 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.587 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2500000000000005e-07 2024-05-06T00:26:59.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265177+0000", 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3440000000000001e-06 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265187+0000", 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755e-05 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265194+0000", 2024-05-06T00:26:59.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3999999999999997e-06 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265713+0000", 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051880999999999997 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265781+0000", 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8140999999999997e-05 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265878+0000", 2024-05-06T00:26:59.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7052999999999994e-05 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.266675+0000", 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00079742599999999995 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271049+0000", 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043737630000000001 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271077+0000", 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.775e-05 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271112+0000", 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4935999999999999e-05 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1186 3.4 e29/18 3:26d09f8b:::rbd_object_map.110cc8beeb04:head v 29'83, mlcod=29'79)", 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.372346+0000", 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1261185920000001, 2024-05-06T00:26:59.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042904459999999998, 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372346+0000", 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372346+0000", 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372347+0000", 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1829999999999999e-06 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372355+0000", 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7100000000000007e-06 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372366+0000", 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1586000000000001e-05 2024-05-06T00:26:59.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372374+0000", 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7489999999999998e-06 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372485+0000", 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011044600000000001 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372504+0000", 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9205e-05 2024-05-06T00:26:59.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.376589+0000", 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040849509999999999 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.376636+0000", 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7615999999999999e-05 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.596 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1187 3.4 e29/18 3:26d09f8b:::rbd_object_map.110cc8beeb04:head v 29'84, mlcod=29'79)", 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.372401+0000", 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.126063512, 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004268272, 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372401+0000", 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372401+0000", 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372401+0000", 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3999999999999997e-07 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372405+0000", 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-06T00:26:59.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372411+0000", 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9310000000000002e-06 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372417+0000", 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7409999999999998e-06 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372788+0000", 2024-05-06T00:26:59.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000370816 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372808+0000", 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9477e-05 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.376642+0000", 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003834086 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.376669+0000", 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7603999999999999e-05 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1194 3.18 3:1ed77c58:::rbd_data.110cc8beeb04.0000000000000032:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2998272~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.372719+0000", 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1257455599999999, 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040630129999999999, 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1194 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372719+0000", 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372719+0000", 2024-05-06T00:26:59.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372720+0000", 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5370000000000001e-06 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372749+0000", 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8232999999999999e-05 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372760+0000", 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1068e-05 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372766+0000", 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2099999999999998e-06 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372811+0000", 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5368000000000003e-05 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.372904+0000", 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2968999999999997e-05 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.373062+0000", 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000157553 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.374192+0000", 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011300850000000001 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.376687+0000", 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024954349999999998 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.376739+0000", 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1795000000000001e-05 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.376782+0000", 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2759999999999997e-05 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.606 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1205 3.4 e29/18 3:26d09f8b:::rbd_object_map.110cc8beeb04:head v 29'87, mlcod=29'83)", 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.1186279589999999, 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043375489999999996, 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:26:59.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:26:59.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379843+0000", 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9789999999999998e-06 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379858+0000", 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4946999999999999e-05 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379865+0000", 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.754e-06 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379942+0000", 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6687000000000006e-05 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379960+0000", 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8357999999999999e-05 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.384139+0000", 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041789239999999997 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.384174+0000", 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5221e-05 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:26:59.612 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:26:59.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:26:59.615 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:26:59.613+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:26:59.715 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:26:59.713+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:26:59.747 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:59.747 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:59.747 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:26:59.748 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:59.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:26:59.817 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:59.817+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:26:59.884 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:26:59.885 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:26:59.885 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:26:59.885 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:26:59.885 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:26:59.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:26:59.940 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:26:59.941+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:00.019 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:00.017+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.463088406000001, 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.684258621000001, 2024-05-06T00:27:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012319788, 2024-05-06T00:27:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45 2024-05-06T00:27:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381231+0000", 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.624e-06 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381243+0000", 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2476e-05 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381263+0000", 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-05 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381278+0000", 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5183e-05 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389490+0000", 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082113770000000006 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389614+0000", 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012416900000000001 2024-05-06T00:27:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389813+0000", 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199321 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391953+0000", 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021402869999999998 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393301+0000", 2024-05-06T00:27:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001347729 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393438+0000", 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136666 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393549+0000", 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110856 2024-05-06T00:27:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.684182818, 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014641942999999999, 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381306+0000", 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.403e-06 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381316+0000", 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-05-06T00:27:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381341+0000", 2024-05-06T00:27:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4372999999999999e-05 2024-05-06T00:27:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381356+0000", 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5207e-05 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390708+0000", 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093519799999999993 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390856+0000", 2024-05-06T00:27:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839200000000001 2024-05-06T00:27:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391064+0000", 2024-05-06T00:27:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000207302 2024-05-06T00:27:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394636+0000", 2024-05-06T00:27:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003572563 2024-05-06T00:27:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395813+0000", 2024-05-06T00:27:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011765910000000001 2024-05-06T00:27:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395885+0000", 2024-05-06T00:27:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2601999999999994e-05 2024-05-06T00:27:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395947+0000", 2024-05-06T00:27:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1183e-05 2024-05-06T00:27:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.684106142000001, 2024-05-06T00:27:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017147241000000001, 2024-05-06T00:27:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 47 2024-05-06T00:27:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381383+0000", 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6199999999999999e-06 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381504+0000", 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120709 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381525+0000", 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1705999999999999e-05 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381541+0000", 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5367e-05 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392051+0000", 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010510442 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392177+0000", 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012575600000000001 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392387+0000", 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020954799999999999 2024-05-06T00:27:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394744+0000", 2024-05-06T00:27:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023578319999999998 2024-05-06T00:27:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398377+0000", 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036325189999999999 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398462+0000", 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4668999999999999e-05 2024-05-06T00:27:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398529+0000", 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7073e-05 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.683906127, 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017264267, 2024-05-06T00:27:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 48 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381583+0000", 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7069999999999999e-06 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381594+0000", 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1365999999999999e-05 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381618+0000", 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4014000000000001e-05 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381639+0000", 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0616e-05 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393664+0000", 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012025179 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393804+0000", 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014017599999999999 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393998+0000", 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019390499999999999 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398573+0000", 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045742789999999997 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398683+0000", 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110136 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398786+0000", 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103259 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398846+0000", 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9630000000000003e-05 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.683814863999999, 2024-05-06T00:27:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018281285000000001, 2024-05-06T00:27:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 49 2024-05-06T00:27:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381674+0000", 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.559e-06 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381685+0000", 2024-05-06T00:27:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1049999999999999e-05 2024-05-06T00:27:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381705+0000", 2024-05-06T00:27:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0227000000000001e-05 2024-05-06T00:27:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381728+0000", 2024-05-06T00:27:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2467e-05 2024-05-06T00:27:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394829+0000", 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013101358 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394953+0000", 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012340499999999999 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395145+0000", 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019252900000000001 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399714+0000", 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045683620000000003 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399788+0000", 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4719000000000001e-05 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399871+0000", 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3092999999999999e-05 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399954+0000", 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2515999999999994e-05 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.683725185, 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020991408, 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 50 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381764+0000", 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381894+0000", 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012972600000000001 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381927+0000", 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3399000000000003e-05 2024-05-06T00:27:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381938+0000", 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0677e-05 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396032+0000", 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014094257000000001 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396199+0000", 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016722899999999999 2024-05-06T00:27:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396415+0000", 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021597200000000001 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.400244+0000", 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038293009999999998 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402502+0000", 2024-05-06T00:27:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002257972 2024-05-06T00:27:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402645+0000", 2024-05-06T00:27:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014225199999999999 2024-05-06T00:27:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402754+0000", 2024-05-06T00:27:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010917300000000001 2024-05-06T00:27:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.683527810000001, 2024-05-06T00:27:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021178922999999999, 2024-05-06T00:27:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 51 2024-05-06T00:27:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381961+0000", 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381968+0000", 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.007e-06 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381979+0000", 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1197e-05 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381987+0000", 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9629999999999998e-06 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397517+0000", 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015529589999999999 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397635+0000", 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011846899999999999 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397830+0000", 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019498200000000001 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402867+0000", 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050367240000000002 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403002+0000", 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134609 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403082+0000", 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0666000000000003e-05 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403139+0000", 2024-05-06T00:27:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6462999999999998e-05 2024-05-06T00:27:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.683487053, 2024-05-06T00:27:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021264655, 2024-05-06T00:27:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-05-06T00:27:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382001+0000", 2024-05-06T00:27:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7300000000000005e-07 2024-05-06T00:27:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382007+0000", 2024-05-06T00:27:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000004e-06 2024-05-06T00:27:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382017+0000", 2024-05-06T00:27:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9799999999999993e-06 2024-05-06T00:27:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382029+0000", 2024-05-06T00:27:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1420000000000001e-05 2024-05-06T00:27:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398910+0000", 2024-05-06T00:27:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016881075999999998 2024-05-06T00:27:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399069+0000", 2024-05-06T00:27:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015880199999999999 2024-05-06T00:27:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399327+0000", 2024-05-06T00:27:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025800200000000001 2024-05-06T00:27:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402970+0000", 2024-05-06T00:27:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003643725 2024-05-06T00:27:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403155+0000", 2024-05-06T00:27:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018459100000000001 2024-05-06T00:27:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403221+0000", 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6106999999999996e-05 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403265+0000", 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4135999999999999e-05 2024-05-06T00:27:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.479889735, 2024-05-06T00:27:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012456859000000001, 2024-05-06T00:27:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-05-06T00:27:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585599+0000", 2024-05-06T00:27:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5899999999999995e-07 2024-05-06T00:27:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585692+0000", 2024-05-06T00:27:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3398000000000003e-05 2024-05-06T00:27:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585704+0000", 2024-05-06T00:27:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2318e-05 2024-05-06T00:27:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585713+0000", 2024-05-06T00:27:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7649999999999999e-06 2024-05-06T00:27:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593668+0000", 2024-05-06T00:27:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079544980000000008 2024-05-06T00:27:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593723+0000", 2024-05-06T00:27:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5914e-05 2024-05-06T00:27:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593815+0000", 2024-05-06T00:27:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1583999999999996e-05 2024-05-06T00:27:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595554+0000", 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017386190000000001 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.597968+0000", 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024139500000000002 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598008+0000", 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9962000000000001e-05 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598055+0000", 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7092000000000002e-05 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.479757477, 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013009979, 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 76 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585731+0000", 2024-05-06T00:27:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0999999999999997e-07 2024-05-06T00:27:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585738+0000", 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6059999999999999e-06 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585752+0000", 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4248e-05 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585762+0000", 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8660000000000001e-06 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594480+0000", 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087185100000000005 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594616+0000", 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135375 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594806+0000", 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019038099999999999 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595986+0000", 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011804039999999999 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598676+0000", 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026891889999999998 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598710+0000", 2024-05-06T00:27:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4471999999999998e-05 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598740+0000", 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0018e-05 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:401 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 786432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2562712349999998, 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012526862999999999, 2024-05-06T00:27:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 401 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809218+0000", 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809224+0000", 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.037e-06 2024-05-06T00:27:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809242+0000", 2024-05-06T00:27:00.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:00.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809256+0000", 2024-05-06T00:27:00.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4027e-05 2024-05-06T00:27:00.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821381+0000", 2024-05-06T00:27:00.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012125268999999999 2024-05-06T00:27:00.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821510+0000", 2024-05-06T00:27:00.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129219 2024-05-06T00:27:00.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821743+0000", 2024-05-06T00:27:00.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023277500000000001 2024-05-06T00:27:00.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.166 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.167 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:402 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 1703936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.167 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:00.167 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2562092820000004, 2024-05-06T00:27:00.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013133265, 2024-05-06T00:27:00.167 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.167 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.168 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.168 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 402 2024-05-06T00:27:00.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.169 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:00.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:00.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809279+0000", 2024-05-06T00:27:00.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:27:00.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809285+0000", 2024-05-06T00:27:00.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:27:00.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809303+0000", 2024-05-06T00:27:00.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7983e-05 2024-05-06T00:27:00.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809317+0000", 2024-05-06T00:27:00.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3856000000000001e-05 2024-05-06T00:27:00.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821841+0000", 2024-05-06T00:27:00.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012524400999999999 2024-05-06T00:27:00.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822066+0000", 2024-05-06T00:27:00.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022526100000000001 2024-05-06T00:27:00.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822412+0000", 2024-05-06T00:27:00.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034507400000000001 2024-05-06T00:27:00.176 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.176 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.176 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.177 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:403 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 2621440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.177 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:00.177 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2561532, 2024-05-06T00:27:00.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013546829999999999, 2024-05-06T00:27:00.177 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.177 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.178 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 403 2024-05-06T00:27:00.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.178 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:00.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:00.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809335+0000", 2024-05-06T00:27:00.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1200000000000001e-06 2024-05-06T00:27:00.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809341+0000", 2024-05-06T00:27:00.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9200000000000001e-06 2024-05-06T00:27:00.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809360+0000", 2024-05-06T00:27:00.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8184000000000001e-05 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809373+0000", 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822550+0000", 2024-05-06T00:27:00.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013176998000000001 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822703+0000", 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015292600000000001 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822881+0000", 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017814600000000001 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:404 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 3538944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2560537700000003, 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013816116999999999, 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 404 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809435+0000", 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4300000000000001e-06 2024-05-06T00:27:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809441+0000", 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9730000000000002e-06 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809461+0000", 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9786e-05 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809498+0000", 2024-05-06T00:27:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6706999999999999e-05 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822957+0000", 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013459296000000001 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823087+0000", 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054200000000001 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823250+0000", 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016238299999999999 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:405 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 4456448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2559681070000002, 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014023473999999999, 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 405 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809521+0000", 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3629999999999999e-06 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809527+0000", 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1369999999999999e-06 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809544+0000", 2024-05-06T00:27:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7516e-05 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809559+0000", 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.455e-05 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823295+0000", 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013736474 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823397+0000", 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010108899999999999 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823543+0000", 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014634500000000001 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:406 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 5373952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2559096849999998, 2024-05-06T00:27:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01426091, 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 406 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809579+0000", 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.077e-06 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809584+0000", 2024-05-06T00:27:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3530000000000002e-06 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809604+0000", 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9962999999999998e-05 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809632+0000", 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7710000000000001e-05 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823589+0000", 2024-05-06T00:27:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013957476 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823695+0000", 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010535900000000001 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823839+0000", 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143972 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:407 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 6291456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2558356479999997, 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014464551000000001, 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 407 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809653+0000", 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2330000000000001e-06 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809659+0000", 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0170000000000002e-06 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809678+0000", 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8703e-05 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809692+0000", 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4326e-05 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823880+0000", 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014187936999999999 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823982+0000", 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102266 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824116+0000", 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013406900000000001 2024-05-06T00:27:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:408 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.255759619, 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014667517, 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 408 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809729+0000", 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5430000000000001e-06 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809749+0000", 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9227000000000001e-05 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809769+0000", 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0556000000000001e-05 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809784+0000", 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4507e-05 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824158+0000", 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014374418999999999 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824258+0000", 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100209 2024-05-06T00:27:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824395+0000", 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013705600000000001 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:409 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2556839929999999, 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014988434, 2024-05-06T00:27:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 409 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809805+0000", 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.917e-06 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809812+0000", 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5930000000000002e-06 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809830+0000", 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8159e-05 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809844+0000", 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3764000000000001e-05 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824478+0000", 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014633703 2024-05-06T00:27:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824611+0000", 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132893 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824792+0000", 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018140499999999999 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.211 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:00.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:00.213 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:00.213+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:00.312 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:00.313+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:00.382 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:00.382 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:00.382 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:00.383 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:00.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:00.414 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:00.413+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:00.483 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone3 from 0xe6a7400 (overlap 0xdca00) to 0x0 2024-05-06T00:27:00.515 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:00.513+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:00.561 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:00.561 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:00.575 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:00.615 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:00.613+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:00.694 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:00.694 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:9 3.1a 3:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.980331464000001, 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084089950000000007, 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:00.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 9 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715203+0000", 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715219+0000", 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6577000000000001e-05 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715254+0000", 2024-05-06T00:27:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5308999999999997e-05 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715279+0000", 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4902999999999999e-05 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715445+0000", 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016536900000000001 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:00.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715456+0000", 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1037e-05 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.719765+0000", 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043095060000000003 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.720002+0000", 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000236596 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.720369+0000", 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036678999999999998 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.722404+0000", 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002035121 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723446+0000", 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010426249999999999 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723534+0000", 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7423000000000006e-05 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723607+0000", 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3195999999999995e-05 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:545 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'48, mlcod=23'46)", 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.014250569, 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004580738, 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681281+0000", 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681293+0000", 2024-05-06T00:27:00.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681321+0000", 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8073e-05 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681342+0000", 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1192999999999999e-05 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681467+0000", 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000124976 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681503+0000", 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6207000000000001e-05 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685774+0000", 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042710869999999998 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685860+0000", 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5228999999999993e-05 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:632 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.957803928000001, 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073037140000000002, 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.706 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 632 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737728+0000", 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6699999999999998e-06 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737738+0000", 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0263e-05 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737764+0000", 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5584000000000001e-05 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737781+0000", 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6849000000000001e-05 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737901+0000", 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012037299999999999 2024-05-06T00:27:00.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737912+0000", 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0343000000000001e-05 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.744738+0000", 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068260940000000004 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.744863+0000", 2024-05-06T00:27:00.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125139 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745029+0000", 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016639899999999999 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:342 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.955555971999999, 2024-05-06T00:27:00.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053952979999999998, 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 342 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739976+0000", 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.88e-06 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739988+0000", 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1440000000000001e-05 2024-05-06T00:27:00.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740019+0000", 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0942999999999998e-05 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740034+0000", 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4766000000000001e-05 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740219+0000", 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185069 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740229+0000", 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0890000000000001e-05 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745083+0000", 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048533760000000004 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745208+0000", 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012539499999999999 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745369+0000", 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016053899999999999 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:852 3.1b e24/18 3:da0a6ebf:::rbd_data.110c17d674e9.0000000000000003:head v 24'68, mlcod=24'66)", 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:27:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.539195926, 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041048589999999998, 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:27:00.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156335+0000", 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0580000000000001e-06 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156365+0000", 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0719999999999997e-05 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156384+0000", 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8715e-05 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156395+0000", 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0597999999999999e-05 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156495+0000", 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100055 2024-05-06T00:27:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156527+0000", 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2458e-05 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.160390+0000", 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038625040000000001 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.160438+0000", 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8751000000000001e-05 2024-05-06T00:27:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:303 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2401792~22528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9249447819999999, 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057218729999999997, 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 303 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770589+0000", 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9099999999999998e-06 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770633+0000", 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4326999999999998e-05 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770670+0000", 2024-05-06T00:27:00.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7101999999999998e-05 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770692+0000", 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1512999999999999e-05 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770851+0000", 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015903600000000001 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.771219+0000", 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0003681 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.771608+0000", 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038980699999999999 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.773584+0000", 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019750890000000002 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776110+0000", 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025268669999999999 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776236+0000", 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125726 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776307+0000", 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0395999999999995e-05 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:325 3.9 3:90865cbe:::rbd_data.110cc8beeb04.0000000000000015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9026428439999998, 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0077305209999999997, 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 325 2024-05-06T00:27:00.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792891+0000", 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9269999999999998e-06 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792905+0000", 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4926000000000001e-05 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792989+0000", 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3999999999999995e-05 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793013+0000", 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3856e-05 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795543+0000", 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025296369999999999 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795768+0000", 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022517500000000001 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.796053+0000", 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028487500000000003 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798879+0000", 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028255210000000001 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800464+0000", 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001585218 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800548+0000", 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4635999999999994e-05 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800617+0000", 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8750000000000004e-05 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:324 3.19 e29/18 3:99a6ca97:::rbd_data.110cc8beeb04.0000000000000013:head v 29'64, mlcod=28'62)", 2024-05-06T00:27:00.732 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9023909489999999, 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052824120000000002, 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793140+0000", 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3799999999999999e-06 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793152+0000", 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2046e-05 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793186+0000", 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3646999999999998e-05 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793202+0000", 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6220000000000001e-05 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794514+0000", 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001312353 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794578+0000", 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3476e-05 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798323+0000", 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037458109999999999 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798421+0000", 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7479000000000006e-05 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.736 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:318 3.14 e29/18 3:29721031:::rbd_data.110cc8beeb04.000000000000000a:head v 29'2, mlcod=0'0)", 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.793831+0000", 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.901698798, 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042763669999999997, 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793831+0000", 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793831+0000", 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793831+0000", 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1699999999999997e-07 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793842+0000", 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0587e-05 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793874+0000", 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2419000000000003e-05 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793894+0000", 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9117e-05 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794258+0000", 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036420400000000003 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794311+0000", 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2748999999999999e-05 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797975+0000", 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003664072 2024-05-06T00:27:00.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798107+0000", 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132402 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:327 3.11 e29/18 3:8dd1a01a:::rbd_data.110cc8beeb04.0000000000000016:head v 29'2, mlcod=0'0)", 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.793945+0000", 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9015843190000004, 2024-05-06T00:27:00.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048477959999999997, 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793945+0000", 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793945+0000", 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793946+0000", 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7399999999999991e-07 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793956+0000", 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9259999999999995e-06 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793976+0000", 2024-05-06T00:27:00.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0214e-05 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793991+0000", 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5116000000000001e-05 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794964+0000", 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00097236200000000005 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795013+0000", 2024-05-06T00:27:00.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9339999999999999e-05 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798426+0000", 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034130599999999999 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798793+0000", 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036680399999999998 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:319 3.1d e29/18 3:b8eceabe:::rbd_data.110cc8beeb04.000000000000000e:head v 29'2, mlcod=0'0)", 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.794131+0000", 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9013986369999998, 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041470839999999997, 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794131+0000", 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794131+0000", 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794132+0000", 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8400000000000003e-07 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794141+0000", 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0699999999999996e-06 2024-05-06T00:27:00.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794164+0000", 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3689e-05 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794183+0000", 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8413000000000001e-05 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794393+0000", 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021038200000000001 2024-05-06T00:27:00.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794445+0000", 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2309999999999999e-05 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798172+0000", 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037265739999999999 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798278+0000", 2024-05-06T00:27:00.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105862 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:328 3.19 e29/18 3:9fbbd60a:::rbd_data.110cc8beeb04.000000000000001b:head v 29'65, mlcod=28'62)", 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.794487+0000", 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9010424160000001, 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043724430000000002, 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794487+0000", 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794487+0000", 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794489+0000", 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-05-06T00:27:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794503+0000", 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4583000000000001e-05 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794528+0000", 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.421e-05 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794562+0000", 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4493000000000003e-05 2024-05-06T00:27:00.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795195+0000", 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063313499999999995 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795245+0000", 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0093000000000003e-05 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798797+0000", 2024-05-06T00:27:00.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035519990000000001 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798859+0000", 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2080000000000002e-05 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:343 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.8984532160000001, 2024-05-06T00:27:00.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062599750000000001, 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 343 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797078+0000", 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0700000000000001e-06 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797088+0000", 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6630000000000005e-06 2024-05-06T00:27:00.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797124+0000", 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6186000000000002e-05 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797147+0000", 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2819999999999998e-05 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797305+0000", 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015812199999999999 2024-05-06T00:27:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797480+0000", 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000175172 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797716+0000", 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235419 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.799507+0000", 2024-05-06T00:27:00.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001791089 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803182+0000", 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003674835 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803266+0000", 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3986000000000005e-05 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803336+0000", 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0612999999999996e-05 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:1044 3.15 3:aa19a24e:::rbd_data.110cc8beeb04.0000000000000004:head [stat,truncate 3538944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:56.266770+0000", 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4287590650000004, 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046412349999999996, 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1044 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266770+0000", 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266770+0000", 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266773+0000", 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5569999999999998e-06 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266783+0000", 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0487e-05 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266815+0000", 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1702000000000002e-05 2024-05-06T00:27:00.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266830+0000", 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.523e-05 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266978+0000", 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148119 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.267223+0000", 2024-05-06T00:27:00.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000244505 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.267606+0000", 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000382977 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.269340+0000", 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017341940000000001 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.271231+0000", 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018904219999999999 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.271323+0000", 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2373000000000005e-05 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.271412+0000", 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8669000000000001e-05 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:1275 3.15 3:aa19a24e:::rbd_data.110cc8beeb04.0000000000000004:head [stat,write 1269760~12288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:56.430322+0000", 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.26520733, 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048771090000000001, 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1275 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.766 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430322+0000", 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430322+0000", 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430325+0000", 2024-05-06T00:27:00.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5000000000000002e-06 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430353+0000", 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7963e-05 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430384+0000", 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1730000000000003e-05 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430399+0000", 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5046e-05 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430507+0000", 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107923 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430690+0000", 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018320500000000001 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:00.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.431112+0000", 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042198600000000001 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.432612+0000", 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014998419999999999 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.435000+0000", 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023872419999999999 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.435086+0000", 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6879000000000006e-05 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.435199+0000", 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112793 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2565 3.15 3:aa19a24e:::rbd_data.110cc8beeb04.0000000000000004:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:00.771 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.500103+0000", 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.19542626799999999, 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043273340000000004, 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2565 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500103+0000", 2024-05-06T00:27:00.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500103+0000", 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500105+0000", 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500115+0000", 2024-05-06T00:27:00.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9189999999999994e-06 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500133+0000", 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7895000000000001e-05 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500143+0000", 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7270000000000002e-06 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500213+0000", 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0807000000000002e-05 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500298+0000", 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4845999999999994e-05 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500464+0000", 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016600600000000001 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.501604+0000", 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001139539 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504308+0000", 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027045580000000001 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504375+0000", 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6915000000000006e-05 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504431+0000", 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5191000000000001e-05 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2577 3.15 3:afc90aa8:::rbd_data.110cc8beeb04.0000000000000023:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.19105777199999999, 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056316919999999998, 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.777 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2577 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504473+0000", 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2619999999999999e-06 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504479+0000", 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8080000000000001e-06 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504494+0000", 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5356999999999999e-05 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504504+0000", 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6739999999999992e-06 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504566+0000", 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2030000000000001e-05 2024-05-06T00:27:00.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504653+0000", 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6718000000000006e-05 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504831+0000", 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017891900000000001 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.506697+0000", 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018657529999999999 2024-05-06T00:27:00.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510025+0000", 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003327909 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510063+0000", 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8243e-05 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510103+0000", 2024-05-06T00:27:00.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0018999999999997e-05 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2578 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.190997316, 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082191650000000005, 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:00.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2578 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504533+0000", 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8700000000000005e-07 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504539+0000", 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5509999999999996e-06 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504551+0000", 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1319e-05 2024-05-06T00:27:00.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504559+0000", 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5720000000000003e-06 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504632+0000", 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2531000000000001e-05 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504726+0000", 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4475000000000006e-05 2024-05-06T00:27:00.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504935+0000", 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020863699999999999 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.506757+0000", 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001822471 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512670+0000", 2024-05-06T00:27:00.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059127989999999998 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512717+0000", 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6320999999999999e-05 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512751+0000", 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4702000000000001e-05 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:2576 3.a e32/18 3:5029607f:::rbd_data.110cc8beeb04.0000000000000017:head v 32'20, mlcod=29'17)", 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.19003175899999999, 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063083260000000004, 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505499+0000", 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0610000000000001e-06 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505508+0000", 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2310000000000002e-06 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505522+0000", 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4419999999999999e-05 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505532+0000", 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2220000000000008e-06 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505599+0000", 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7485999999999997e-05 2024-05-06T00:27:00.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505619+0000", 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9995000000000001e-05 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.511754+0000", 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061349300000000002 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.511806+0000", 2024-05-06T00:27:00.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1981000000000003e-05 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:2581 3.1b e32/18 3:d89117cf:::rbd_data.110cc8beeb04.0000000000000033:head v 32'120, mlcod=29'116)", 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.509841+0000", 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.18568847999999999, 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046528209999999997, 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509841+0000", 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509841+0000", 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509842+0000", 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0210000000000001e-06 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509851+0000", 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.969e-06 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509866+0000", 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4915e-05 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509875+0000", 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3449999999999995e-06 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509955+0000", 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0179999999999995e-05 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510016+0000", 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0207999999999997e-05 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.514458+0000", 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044422430000000002 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:00.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.514494+0000", 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5939999999999998e-05 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:00.797 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:00.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:00.799 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:00.797+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:00.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:00.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:27:00.900 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:00.897+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:00.930 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:00.930 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:00.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:00.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:00.950 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:00.963 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-05-06T00:27:00.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:00.967 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:00.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:00.984 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:01.001 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:01.001+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:01.001 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:01.020 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:01.039 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:01.040 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:01.066 INFO:teuthology.orchestra.run.smithi186.stdout:632 punch from 0x3b81800 to 0x3b86000, (0x4800 bytes) 2024-05-06T00:27:01.068 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:01.068 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:01.081 INFO:teuthology.orchestra.run.smithi186.stdout:633 read 0xe547200 thru 0xe551cc5 (0xaac6 bytes) 2024-05-06T00:27:01.081 INFO:teuthology.orchestra.run.smithi186.stdout:634 read 0x67ab200 thru 0x67b037c (0x517d bytes) 2024-05-06T00:27:01.082 INFO:teuthology.orchestra.run.smithi186.stdout:635 write 0xb930800 thru 0xb938735 (0x7f36 bytes) 2024-05-06T00:27:01.082 INFO:teuthology.orchestra.run.smithi186.stdout:636 write 0xdef1a00 thru 0xdef2357 (0x958 bytes) 2024-05-06T00:27:01.082 INFO:teuthology.orchestra.run.smithi186.stdout:637 write 0x792a800 thru 0x7939c81 (0xf482 bytes) 2024-05-06T00:27:01.082 INFO:teuthology.orchestra.run.smithi186.stdout:638 read 0x5244800 thru 0x524ebb6 (0xa3b7 bytes) 2024-05-06T00:27:01.083 INFO:teuthology.orchestra.run.smithi186.stdout:639 write 0x8647600 thru 0x864d862 (0x6263 bytes) 2024-05-06T00:27:01.083 INFO:teuthology.orchestra.run.smithi186.stdout:640 clone 5 order 25 su 2097152 sc 12 2024-05-06T00:27:01.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:01.102 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:01.101+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:01.203 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:01.201+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:01.251 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:01.252 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:01.266 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:01.314 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:01.313+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:01.382 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:01.382 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:01.394 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:01.404 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:01.405+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:01.505 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:01.501+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:01.559 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:01.559 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:01.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:01.618 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:01.617+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:01.707 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:01.709+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:01.740 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:01.740 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:01.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:01.807 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:01.805+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:01.815 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:27:01.877 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:01.877 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:01.890 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:01.909 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:01.909+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:01.992 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone4 from 0xe6a7400 (overlap 0xe6a7400) to 0x5c42e00 2024-05-06T00:27:02.009 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:02.009+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:02.061 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:02.062 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:02.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:02.126 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:02.125+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:02.212 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:02.209+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:02.250 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:02.250 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:02.265 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:02.312 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:02.313+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:02.413 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:02.413+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:02.431 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:02.431 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:02.446 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:02.534 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:02.533+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:02.614 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:02.613+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:02.617 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:02.617 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:02.632 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:02.660 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-05-06T00:27:02.716 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:02.717+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:02.726 INFO:teuthology.orchestra.run.smithi186.stdout:641 punch from 0x4e9ae00 to 0x4e9c400, (0x1600 bytes) 2024-05-06T00:27:02.804 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:02.804 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:02.814 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:02.814 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:27:02.815 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-05-06T00:27:02.815 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-06T00:27:02.815 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-06T00:27:02.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-05-06T00:27:02.817 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:02.817+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:02.817 INFO:teuthology.orchestra.run.smithi186.stdout:644 read 0x8b4ba00 thru 0x8b56efb (0xb4fc bytes) 2024-05-06T00:27:02.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:02.820 INFO:teuthology.orchestra.run.smithi186.stdout:645 read 0x7366400 thru 0x7372aba (0xc6bb bytes) 2024-05-06T00:27:02.918 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:02.917+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:02.929 INFO:teuthology.orchestra.run.smithi186.stdout:647 trunc from 0xe6a7400 to 0x3ed1000 2024-05-06T00:27:02.948 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:02.948 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-06T00:27:02.948 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:02.993 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:02.994 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:03.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:03.018 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:03.017+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:03.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:03.120 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:03.117+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:03.167 INFO:teuthology.orchestra.run.smithi186.stdout:650 trunc from 0x3ed1000 to 0x95be400 2024-05-06T00:27:03.178 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:03.178 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:03.185 INFO:teuthology.orchestra.run.smithi186.stdout:651 write 0x1fac400 thru 0x1fb7494 (0xb095 bytes) 2024-05-06T00:27:03.186 INFO:teuthology.orchestra.run.smithi186.stdout:656 read 0x8af1600 thru 0x8afb9ef (0xa3f0 bytes) 2024-05-06T00:27:03.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:03.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:03.216 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:03.220 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:03.217+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:03.221 INFO:teuthology.orchestra.run.smithi186.stdout:657 trunc from 0x95be400 to 0x270fe00 2024-05-06T00:27:03.233 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:03.244 INFO:teuthology.orchestra.run.smithi186.stdout:658 write 0x1a24e00 thru 0x1a2cee0 (0x80e1 bytes) 2024-05-06T00:27:03.251 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:03.251 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:03.267 INFO:teuthology.orchestra.run.smithi186.stdout:660 write 0x4b69000 thru 0x4b74210 (0xb211 bytes) 2024-05-06T00:27:03.268 INFO:teuthology.orchestra.run.smithi186.stdout:661 read 0x2b9a400 thru 0x2baa293 (0xfe94 bytes) 2024-05-06T00:27:03.272 INFO:teuthology.orchestra.run.smithi186.stdout:662 trunc from 0x4b74400 to 0x57df200 2024-05-06T00:27:03.272 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:03.272 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:03.289 INFO:teuthology.orchestra.run.smithi186.stdout:666 read 0x13d9a00 thru 0x13e1eef (0x84f0 bytes) 2024-05-06T00:27:03.291 INFO:teuthology.orchestra.run.smithi186.stdout:667 punch from 0x1886a00 to 0x188be00, (0x5400 bytes) 2024-05-06T00:27:03.291 INFO:teuthology.orchestra.run.smithi186.stdout:670 punch from 0x2f64400 to 0x2f6b200, (0x6e00 bytes) 2024-05-06T00:27:03.291 INFO:teuthology.orchestra.run.smithi186.stdout:671 write 0x2947c00 thru 0x294cbe7 (0x4fe8 bytes) 2024-05-06T00:27:03.292 INFO:teuthology.orchestra.run.smithi186.stdout:672 read 0x3230600 thru 0x3239458 (0x8e59 bytes) 2024-05-06T00:27:03.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:03.316 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:03.317 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:03.321 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:03.317+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:03.327 INFO:teuthology.orchestra.run.smithi186.stdout:673 write 0xbe35400 thru 0xbe4395f (0xe560 bytes) 2024-05-06T00:27:03.331 INFO:teuthology.orchestra.run.smithi186.stdout:674 read 0x81cd800 thru 0x81cdcd9 (0x4da bytes) 2024-05-06T00:27:03.331 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.727265559999999, 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:03.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:03.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.948435775, 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012319788, 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.339 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381231+0000", 2024-05-06T00:27:03.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.624e-06 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381243+0000", 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2476e-05 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381263+0000", 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-05 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381278+0000", 2024-05-06T00:27:03.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5183e-05 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389490+0000", 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082113770000000006 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389614+0000", 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012416900000000001 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389813+0000", 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199321 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391953+0000", 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021402869999999998 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393301+0000", 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001347729 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393438+0000", 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136666 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393549+0000", 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110856 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.948359971999999, 2024-05-06T00:27:03.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014641942999999999, 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381306+0000", 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.403e-06 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381316+0000", 2024-05-06T00:27:03.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381341+0000", 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4372999999999999e-05 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381356+0000", 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5207e-05 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390708+0000", 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093519799999999993 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390856+0000", 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839200000000001 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391064+0000", 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000207302 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394636+0000", 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003572563 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395813+0000", 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011765910000000001 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395885+0000", 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2601999999999994e-05 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395947+0000", 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1183e-05 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.948283296, 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017147241000000001, 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.350 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 47 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381383+0000", 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6199999999999999e-06 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381504+0000", 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120709 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381525+0000", 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1705999999999999e-05 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381541+0000", 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5367e-05 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392051+0000", 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010510442 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392177+0000", 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012575600000000001 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392387+0000", 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020954799999999999 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394744+0000", 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023578319999999998 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398377+0000", 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036325189999999999 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398462+0000", 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4668999999999999e-05 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398529+0000", 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7073e-05 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.355 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.948083281000002, 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017264267, 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.356 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 48 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381583+0000", 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7069999999999999e-06 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381594+0000", 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1365999999999999e-05 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381618+0000", 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4014000000000001e-05 2024-05-06T00:27:03.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381639+0000", 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0616e-05 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393664+0000", 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012025179 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393804+0000", 2024-05-06T00:27:03.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014017599999999999 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393998+0000", 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019390499999999999 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398573+0000", 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045742789999999997 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398683+0000", 2024-05-06T00:27:03.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110136 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398786+0000", 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103259 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398846+0000", 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9630000000000003e-05 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.947992018000001, 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018281285000000001, 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 49 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.362 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381674+0000", 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.559e-06 2024-05-06T00:27:03.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381685+0000", 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1049999999999999e-05 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381705+0000", 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0227000000000001e-05 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381728+0000", 2024-05-06T00:27:03.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2467e-05 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394829+0000", 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013101358 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394953+0000", 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012340499999999999 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395145+0000", 2024-05-06T00:27:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019252900000000001 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399714+0000", 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045683620000000003 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399788+0000", 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4719000000000001e-05 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399871+0000", 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3092999999999999e-05 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399954+0000", 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2515999999999994e-05 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:03.367 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.947902339000002, 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020991408, 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 50 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381764+0000", 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381894+0000", 2024-05-06T00:27:03.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012972600000000001 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381927+0000", 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3399000000000003e-05 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381938+0000", 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0677e-05 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396032+0000", 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014094257000000001 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396199+0000", 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016722899999999999 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396415+0000", 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021597200000000001 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.400244+0000", 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038293009999999998 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402502+0000", 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002257972 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402645+0000", 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014225199999999999 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402754+0000", 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010917300000000001 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.947704964000003, 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021178922999999999, 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.373 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 51 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381961+0000", 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381968+0000", 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.007e-06 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381979+0000", 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1197e-05 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381987+0000", 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9629999999999998e-06 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397517+0000", 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015529589999999999 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397635+0000", 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011846899999999999 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397830+0000", 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019498200000000001 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402867+0000", 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050367240000000002 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403002+0000", 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134609 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403082+0000", 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0666000000000003e-05 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403139+0000", 2024-05-06T00:27:03.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6462999999999998e-05 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.947664207000003, 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021264655, 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.379 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382001+0000", 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7300000000000005e-07 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382007+0000", 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000004e-06 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382017+0000", 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9799999999999993e-06 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382029+0000", 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1420000000000001e-05 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398910+0000", 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016881075999999998 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399069+0000", 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015880199999999999 2024-05-06T00:27:03.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399327+0000", 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025800200000000001 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402970+0000", 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003643725 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403155+0000", 2024-05-06T00:27:03.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018459100000000001 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403221+0000", 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6106999999999996e-05 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403265+0000", 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4135999999999999e-05 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.744066889000003, 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012456859000000001, 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585599+0000", 2024-05-06T00:27:03.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5899999999999995e-07 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585692+0000", 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3398000000000003e-05 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585704+0000", 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2318e-05 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585713+0000", 2024-05-06T00:27:03.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7649999999999999e-06 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593668+0000", 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079544980000000008 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593723+0000", 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5914e-05 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593815+0000", 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1583999999999996e-05 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595554+0000", 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017386190000000001 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.597968+0000", 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024139500000000002 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598008+0000", 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9962000000000001e-05 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598055+0000", 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7092000000000002e-05 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:03.390 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.743934631000002, 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013009979, 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 76 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:03.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585731+0000", 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0999999999999997e-07 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585738+0000", 2024-05-06T00:27:03.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6059999999999999e-06 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585752+0000", 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4248e-05 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585762+0000", 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8660000000000001e-06 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594480+0000", 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087185100000000005 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594616+0000", 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135375 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594806+0000", 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019038099999999999 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595986+0000", 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011804039999999999 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598676+0000", 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026891889999999998 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598710+0000", 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4471999999999998e-05 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598740+0000", 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0018e-05 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:401 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 786432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5204483890000002, 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012526862999999999, 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.396 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 401 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809218+0000", 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809224+0000", 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.037e-06 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809242+0000", 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809256+0000", 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4027e-05 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821381+0000", 2024-05-06T00:27:03.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012125268999999999 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821510+0000", 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129219 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821743+0000", 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023277500000000001 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:402 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 1703936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5203864359999999, 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013133265, 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 402 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809279+0000", 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809285+0000", 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809303+0000", 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7983e-05 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809317+0000", 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3856000000000001e-05 2024-05-06T00:27:03.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821841+0000", 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012524400999999999 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822066+0000", 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022526100000000001 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822412+0000", 2024-05-06T00:27:03.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034507400000000001 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:403 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 2621440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5203303540000004, 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013546829999999999, 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 403 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809335+0000", 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1200000000000001e-06 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809341+0000", 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9200000000000001e-06 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809360+0000", 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8184000000000001e-05 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809373+0000", 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822550+0000", 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013176998000000001 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822703+0000", 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015292600000000001 2024-05-06T00:27:03.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822881+0000", 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017814600000000001 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:404 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 3538944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5202309239999998, 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013816116999999999, 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.409 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 404 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809435+0000", 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4300000000000001e-06 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809441+0000", 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9730000000000002e-06 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809461+0000", 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9786e-05 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809498+0000", 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6706999999999999e-05 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822957+0000", 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013459296000000001 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823087+0000", 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054200000000001 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823250+0000", 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016238299999999999 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:405 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 4456448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.413 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5201452609999997, 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014023473999999999, 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 405 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809521+0000", 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3629999999999999e-06 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809527+0000", 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1369999999999999e-06 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809544+0000", 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7516e-05 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809559+0000", 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.455e-05 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823295+0000", 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013736474 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823397+0000", 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010108899999999999 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823543+0000", 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014634500000000001 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:406 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 5373952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5200868390000002, 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01426091, 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 406 2024-05-06T00:27:03.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809579+0000", 2024-05-06T00:27:03.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.077e-06 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809584+0000", 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3530000000000002e-06 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809604+0000", 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9962999999999998e-05 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809632+0000", 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7710000000000001e-05 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823589+0000", 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013957476 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823695+0000", 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010535900000000001 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823839+0000", 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143972 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:407 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 6291456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5200128020000001, 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014464551000000001, 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 407 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809653+0000", 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2330000000000001e-06 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809659+0000", 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0170000000000002e-06 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809678+0000", 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8703e-05 2024-05-06T00:27:03.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809692+0000", 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4326e-05 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823880+0000", 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014187936999999999 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823982+0000", 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102266 2024-05-06T00:27:03.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824116+0000", 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013406900000000001 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:408 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5199367729999995, 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014667517, 2024-05-06T00:27:03.426 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 408 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809729+0000", 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5430000000000001e-06 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809749+0000", 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9227000000000001e-05 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809769+0000", 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0556000000000001e-05 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809784+0000", 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4507e-05 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824158+0000", 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014374418999999999 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824258+0000", 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100209 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824395+0000", 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013705600000000001 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.430 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:409 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5198611470000003, 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014988434, 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 409 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809805+0000", 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.917e-06 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809812+0000", 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5930000000000002e-06 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809830+0000", 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8159e-05 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809844+0000", 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3764000000000001e-05 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824478+0000", 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014633703 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824611+0000", 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132893 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824792+0000", 2024-05-06T00:27:03.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018140499999999999 2024-05-06T00:27:03.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.435 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.435 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.435 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:03.437 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:03.433+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:03.438 INFO:teuthology.orchestra.run.smithi186.stdout:676 read 0x2ad2600 thru 0x2add5b9 (0xafba bytes) 2024-05-06T00:27:03.438 INFO:teuthology.orchestra.run.smithi186.stdout:677 read 0x5fa2c00 thru 0x5fa4e6e (0x226f bytes) 2024-05-06T00:27:03.438 INFO:teuthology.orchestra.run.smithi186.stdout:678 read 0x7047600 thru 0x7053f5a (0xc95b bytes) 2024-05-06T00:27:03.438 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0xece5800 2024-05-06T00:27:03.438 INFO:teuthology.orchestra.run.smithi186.stdout:680 trunc from 0xbe43a00 to 0xece5800 2024-05-06T00:27:03.438 INFO:teuthology.orchestra.run.smithi186.stdout:682 trunc from 0xece5800 to 0xd10000 2024-05-06T00:27:03.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:03.471 INFO:teuthology.orchestra.run.smithi186.stdout:683 punch from 0x612e00 to 0x61d600, (0xa800 bytes) 2024-05-06T00:27:03.521 INFO:teuthology.orchestra.run.smithi186.stdout:684 write 0xd8ebc00 thru 0xd8f6b54 (0xaf55 bytes) 2024-05-06T00:27:03.537 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:03.533+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:03.538 INFO:teuthology.orchestra.run.smithi186.stdout:689 trunc from 0xd8f6c00 to 0x6890800 2024-05-06T00:27:03.572 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:03.572 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:03.572 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:03.572 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:03.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:03.587 INFO:teuthology.orchestra.run.smithi186.stdout:690 trunc from 0x6890800 to 0xd90c800 2024-05-06T00:27:03.611 INFO:teuthology.orchestra.run.smithi186.stdout:693 read 0x7073200 thru 0x707cac8 (0x98c9 bytes) 2024-05-06T00:27:03.613 INFO:teuthology.orchestra.run.smithi186.stdout:694 write 0xb5b0000 thru 0xb5bdcc1 (0xdcc2 bytes) 2024-05-06T00:27:03.613 INFO:teuthology.orchestra.run.smithi186.stdout:696 punch from 0x64f5800 to 0x64f7200, (0x1a00 bytes) 2024-05-06T00:27:03.614 INFO:teuthology.orchestra.run.smithi186.stdout:698 read 0x7339200 thru 0x73472e1 (0xe0e2 bytes) 2024-05-06T00:27:03.628 INFO:teuthology.orchestra.run.smithi186.stdout:700 trunc from 0xd90c800 to 0x4ce1800 2024-05-06T00:27:03.638 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:03.637+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:03.657 INFO:teuthology.orchestra.run.smithi186.stdout:703 punch from 0x500800 to 0x504e00, (0x4600 bytes) 2024-05-06T00:27:03.658 INFO:teuthology.orchestra.run.smithi186.stdout:707 write 0xf3a200 thru 0xf488fb (0xe6fc bytes) 2024-05-06T00:27:03.659 INFO:teuthology.orchestra.run.smithi186.stdout:709 read 0xdde600 thru 0xdee1c2 (0xfbc3 bytes) 2024-05-06T00:27:03.659 INFO:teuthology.orchestra.run.smithi186.stdout:711 punch from 0x3957600 to 0x395d200, (0x5c00 bytes) 2024-05-06T00:27:03.660 INFO:teuthology.orchestra.run.smithi186.stdout:713 read 0x2ab600 thru 0x2b2a04 (0x7405 bytes) 2024-05-06T00:27:03.661 INFO:teuthology.orchestra.run.smithi186.stdout:714 punch from 0x3960200 to 0x3969800, (0x9600 bytes) 2024-05-06T00:27:03.661 INFO:teuthology.orchestra.run.smithi186.stdout:716 punch from 0xa0b400 to 0xa0ca00, (0x1600 bytes) 2024-05-06T00:27:03.661 INFO:teuthology.orchestra.run.smithi186.stdout:717 write 0x25a9a00 thru 0x25a9c77 (0x278 bytes) 2024-05-06T00:27:03.662 INFO:teuthology.orchestra.run.smithi186.stdout:721 punch from 0x404d200 to 0x4054000, (0x6e00 bytes) 2024-05-06T00:27:03.662 INFO:teuthology.orchestra.run.smithi186.stdout:723 punch from 0x2607000 to 0x260cc00, (0x5c00 bytes) 2024-05-06T00:27:03.662 INFO:teuthology.orchestra.run.smithi186.stdout:724 punch from 0x42d9a00 to 0x42e2400, (0x8a00 bytes) 2024-05-06T00:27:03.695 INFO:teuthology.orchestra.run.smithi186.stdout:725 write 0x5622200 thru 0x562e06f (0xbe70 bytes) 2024-05-06T00:27:03.699 INFO:teuthology.orchestra.run.smithi186.stdout:726 read 0x1209e00 thru 0x12152ef (0xb4f0 bytes) 2024-05-06T00:27:03.700 INFO:teuthology.orchestra.run.smithi186.stdout:727 punch from 0x2856e00 to 0x2865800, (0xea00 bytes) 2024-05-06T00:27:03.701 INFO:teuthology.orchestra.run.smithi186.stdout:728 read 0x4241800 thru 0x4241fc4 (0x7c5 bytes) 2024-05-06T00:27:03.703 INFO:teuthology.orchestra.run.smithi186.stdout:729 read 0x1db3800 thru 0x1db7d03 (0x4504 bytes) 2024-05-06T00:27:03.716 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:03.716 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:03.717 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:03.717 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:03.717 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:03.727 INFO:teuthology.orchestra.run.smithi186.stdout:730 write 0x5afa800 thru 0x5afbb8e (0x138f bytes) 2024-05-06T00:27:03.731 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:03.738 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:03.737+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:03.742 INFO:teuthology.orchestra.run.smithi186.stdout:731 trunc from 0x5afbc00 to 0xbb2c600 2024-05-06T00:27:03.782 INFO:teuthology.orchestra.run.smithi186.stdout:732 trunc from 0xbb2c600 to 0x14c1c00 2024-05-06T00:27:03.828 INFO:teuthology.orchestra.run.smithi186.stdout:734 read 0x3d400 thru 0x4b5d4 (0xe1d5 bytes) 2024-05-06T00:27:03.832 INFO:teuthology.orchestra.run.smithi186.stdout:735 punch from 0x3fdc00 to 0x409400, (0xb800 bytes) 2024-05-06T00:27:03.839 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:03.837+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:03.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:9 3.1a 3:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.144299207, 2024-05-06T00:27:03.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084089950000000007, 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 9 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:03.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715203+0000", 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715219+0000", 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6577000000000001e-05 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715254+0000", 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5308999999999997e-05 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715279+0000", 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4902999999999999e-05 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715445+0000", 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016536900000000001 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715456+0000", 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1037e-05 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.719765+0000", 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043095060000000003 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.720002+0000", 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000236596 2024-05-06T00:27:03.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.720369+0000", 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036678999999999998 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.722404+0000", 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002035121 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723446+0000", 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010426249999999999 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723534+0000", 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7423000000000006e-05 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723607+0000", 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3195999999999995e-05 2024-05-06T00:27:03.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:545 3.1b e23/18 3:df802a3a:::rbd_object_map.110c17d674e9:head v 23'48, mlcod=23'46)", 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.178218312, 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004580738, 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.867 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:27:03.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681279+0000", 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681281+0000", 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681293+0000", 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681321+0000", 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8073e-05 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681342+0000", 2024-05-06T00:27:03.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1192999999999999e-05 2024-05-06T00:27:03.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681467+0000", 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000124976 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.681503+0000", 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6207000000000001e-05 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685774+0000", 2024-05-06T00:27:03.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042710869999999998 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.685860+0000", 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5228999999999993e-05 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:632 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.121771670999999, 2024-05-06T00:27:03.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073037140000000002, 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 632 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:03.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737728+0000", 2024-05-06T00:27:03.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6699999999999998e-06 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737738+0000", 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0263e-05 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737764+0000", 2024-05-06T00:27:03.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5584000000000001e-05 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737781+0000", 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6849000000000001e-05 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737901+0000", 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012037299999999999 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737912+0000", 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0343000000000001e-05 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.744738+0000", 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068260940000000004 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.744863+0000", 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125139 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745029+0000", 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016639899999999999 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:342 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:27:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.119523715, 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053952979999999998, 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 342 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739976+0000", 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.88e-06 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739988+0000", 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1440000000000001e-05 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740019+0000", 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0942999999999998e-05 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740034+0000", 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4766000000000001e-05 2024-05-06T00:27:03.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740219+0000", 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185069 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740229+0000", 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0890000000000001e-05 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745083+0000", 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048533760000000004 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745208+0000", 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012539499999999999 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745369+0000", 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016053899999999999 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.884 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:852 3.1b e24/18 3:da0a6ebf:::rbd_data.110c17d674e9.0000000000000003:head v 24'68, mlcod=24'66)", 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.703163669, 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041048589999999998, 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156334+0000", 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156335+0000", 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0580000000000001e-06 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156365+0000", 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0719999999999997e-05 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156384+0000", 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8715e-05 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156395+0000", 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0597999999999999e-05 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156495+0000", 2024-05-06T00:27:03.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100055 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.156527+0000", 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2458e-05 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.160390+0000", 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038625040000000001 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:50.160438+0000", 2024-05-06T00:27:03.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8751000000000001e-05 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:303 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2401792~22528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.0889125249999996, 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057218729999999997, 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.890 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 303 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770589+0000", 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9099999999999998e-06 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770633+0000", 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4326999999999998e-05 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770670+0000", 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7101999999999998e-05 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770692+0000", 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1512999999999999e-05 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770851+0000", 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015903600000000001 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.771219+0000", 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0003681 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.771608+0000", 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038980699999999999 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.773584+0000", 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019750890000000002 2024-05-06T00:27:03.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776110+0000", 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025268669999999999 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776236+0000", 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125726 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776307+0000", 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0395999999999995e-05 2024-05-06T00:27:03.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:325 3.9 3:90865cbe:::rbd_data.110cc8beeb04.0000000000000015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.0666105869999996, 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0077305209999999997, 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.896 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 325 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792891+0000", 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9269999999999998e-06 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792905+0000", 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4926000000000001e-05 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792989+0000", 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3999999999999995e-05 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793013+0000", 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3856e-05 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795543+0000", 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025296369999999999 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795768+0000", 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022517500000000001 2024-05-06T00:27:03.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.796053+0000", 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028487500000000003 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798879+0000", 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028255210000000001 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800464+0000", 2024-05-06T00:27:03.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001585218 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800548+0000", 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4635999999999994e-05 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800617+0000", 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8750000000000004e-05 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:324 3.19 e29/18 3:99a6ca97:::rbd_data.110cc8beeb04.0000000000000013:head v 29'64, mlcod=28'62)", 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.0663586919999997, 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052824120000000002, 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:03.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793140+0000", 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3799999999999999e-06 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793152+0000", 2024-05-06T00:27:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2046e-05 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793186+0000", 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3646999999999998e-05 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793202+0000", 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6220000000000001e-05 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794514+0000", 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001312353 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794578+0000", 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3476e-05 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798323+0000", 2024-05-06T00:27:03.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037458109999999999 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798421+0000", 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7479000000000006e-05 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:318 3.14 e29/18 3:29721031:::rbd_data.110cc8beeb04.000000000000000a:head v 29'2, mlcod=0'0)", 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.793831+0000", 2024-05-06T00:27:03.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.0656665410000006, 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042763669999999997, 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793831+0000", 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793831+0000", 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793831+0000", 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1699999999999997e-07 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793842+0000", 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0587e-05 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793874+0000", 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2419000000000003e-05 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793894+0000", 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9117e-05 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794258+0000", 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036420400000000003 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794311+0000", 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2748999999999999e-05 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797975+0000", 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003664072 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798107+0000", 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132402 2024-05-06T00:27:03.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:327 3.11 e29/18 3:8dd1a01a:::rbd_data.110cc8beeb04.0000000000000016:head v 29'2, mlcod=0'0)", 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.793945+0000", 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.0655520620000001, 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048477959999999997, 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793945+0000", 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793945+0000", 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793946+0000", 2024-05-06T00:27:03.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7399999999999991e-07 2024-05-06T00:27:03.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793956+0000", 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9259999999999995e-06 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793976+0000", 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0214e-05 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793991+0000", 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5116000000000001e-05 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794964+0000", 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00097236200000000005 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795013+0000", 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9339999999999999e-05 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798426+0000", 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034130599999999999 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798793+0000", 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036680399999999998 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:319 3.1d e29/18 3:b8eceabe:::rbd_data.110cc8beeb04.000000000000000e:head v 29'2, mlcod=0'0)", 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.794131+0000", 2024-05-06T00:27:03.916 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.0653663800000004, 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041470839999999997, 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794131+0000", 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794131+0000", 2024-05-06T00:27:03.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794132+0000", 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8400000000000003e-07 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794141+0000", 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0699999999999996e-06 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794164+0000", 2024-05-06T00:27:03.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3689e-05 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794183+0000", 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8413000000000001e-05 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794393+0000", 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021038200000000001 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794445+0000", 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2309999999999999e-05 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798172+0000", 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037265739999999999 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798278+0000", 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105862 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:328 3.19 e29/18 3:9fbbd60a:::rbd_data.110cc8beeb04.000000000000001b:head v 29'65, mlcod=28'62)", 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.794487+0000", 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.0650101589999998, 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043724430000000002, 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794487+0000", 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794487+0000", 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794489+0000", 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794503+0000", 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4583000000000001e-05 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794528+0000", 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.421e-05 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794562+0000", 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4493000000000003e-05 2024-05-06T00:27:03.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795195+0000", 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063313499999999995 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795245+0000", 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0093000000000003e-05 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798797+0000", 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035519990000000001 2024-05-06T00:27:03.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798859+0000", 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2080000000000002e-05 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:343 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.0624209590000007, 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062599750000000001, 2024-05-06T00:27:03.925 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 343 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:03.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797078+0000", 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0700000000000001e-06 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797088+0000", 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6630000000000005e-06 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797124+0000", 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6186000000000002e-05 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797147+0000", 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2819999999999998e-05 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797305+0000", 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015812199999999999 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797480+0000", 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000175172 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797716+0000", 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235419 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.799507+0000", 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001791089 2024-05-06T00:27:03.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803182+0000", 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003674835 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803266+0000", 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3986000000000005e-05 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803336+0000", 2024-05-06T00:27:03.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0612999999999996e-05 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:1044 3.15 3:aa19a24e:::rbd_data.110cc8beeb04.0000000000000004:head [stat,truncate 3538944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:56.266770+0000", 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5927268080000001, 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046412349999999996, 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1044 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266770+0000", 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266770+0000", 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266773+0000", 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5569999999999998e-06 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266783+0000", 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0487e-05 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266815+0000", 2024-05-06T00:27:03.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1702000000000002e-05 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266830+0000", 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.523e-05 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.266978+0000", 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148119 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.267223+0000", 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000244505 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.267606+0000", 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000382977 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.269340+0000", 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017341940000000001 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.271231+0000", 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018904219999999999 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.271323+0000", 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2373000000000005e-05 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.271412+0000", 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8669000000000001e-05 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:1275 3.15 3:aa19a24e:::rbd_data.110cc8beeb04.0000000000000004:head [stat,write 1269760~12288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:56.430322+0000", 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.4291750729999997, 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048771090000000001, 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.938 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1275 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430322+0000", 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430322+0000", 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430325+0000", 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5000000000000002e-06 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430353+0000", 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7963e-05 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430384+0000", 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1730000000000003e-05 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430399+0000", 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5046e-05 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430507+0000", 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107923 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.430690+0000", 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018320500000000001 2024-05-06T00:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.431112+0000", 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042198600000000001 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.432612+0000", 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014998419999999999 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.435000+0000", 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023872419999999999 2024-05-06T00:27:03.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.435086+0000", 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6879000000000006e-05 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.435199+0000", 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112793 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2565 3.15 3:aa19a24e:::rbd_data.110cc8beeb04.0000000000000004:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.500103+0000", 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.359394011, 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043273340000000004, 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2565 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500103+0000", 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500103+0000", 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500105+0000", 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-05-06T00:27:03.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500115+0000", 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9189999999999994e-06 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500133+0000", 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7895000000000001e-05 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500143+0000", 2024-05-06T00:27:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7270000000000002e-06 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500213+0000", 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0807000000000002e-05 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500298+0000", 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4845999999999994e-05 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:03.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.500464+0000", 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016600600000000001 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.501604+0000", 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001139539 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504308+0000", 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027045580000000001 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504375+0000", 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6915000000000006e-05 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504431+0000", 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5191000000000001e-05 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.949 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2577 3.15 3:afc90aa8:::rbd_data.110cc8beeb04.0000000000000023:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.3550255149999999, 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056316919999999998, 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2577 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504473+0000", 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2619999999999999e-06 2024-05-06T00:27:03.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504479+0000", 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8080000000000001e-06 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504494+0000", 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5356999999999999e-05 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504504+0000", 2024-05-06T00:27:03.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6739999999999992e-06 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504566+0000", 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2030000000000001e-05 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504653+0000", 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6718000000000006e-05 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504831+0000", 2024-05-06T00:27:03.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017891900000000001 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.506697+0000", 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018657529999999999 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510025+0000", 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003327909 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510063+0000", 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8243e-05 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510103+0000", 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0018999999999997e-05 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2578 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:03.955 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.354965059, 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082191650000000005, 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2578 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:03.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504533+0000", 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8700000000000005e-07 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504539+0000", 2024-05-06T00:27:03.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5509999999999996e-06 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504551+0000", 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1319e-05 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504559+0000", 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5720000000000003e-06 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504632+0000", 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2531000000000001e-05 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504726+0000", 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4475000000000006e-05 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504935+0000", 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020863699999999999 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.506757+0000", 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001822471 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512670+0000", 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059127989999999998 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512717+0000", 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6320999999999999e-05 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512751+0000", 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4702000000000001e-05 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:2576 3.a e32/18 3:5029607f:::rbd_data.110cc8beeb04.0000000000000017:head v 32'20, mlcod=29'17)", 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.3539995020000002, 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063083260000000004, 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.961 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505499+0000", 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0610000000000001e-06 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505508+0000", 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2310000000000002e-06 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505522+0000", 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4419999999999999e-05 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505532+0000", 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2220000000000008e-06 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505599+0000", 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7485999999999997e-05 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505619+0000", 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9995000000000001e-05 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.511754+0000", 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061349300000000002 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.511806+0000", 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1981000000000003e-05 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:2581 3.1b e32/18 3:d89117cf:::rbd_data.110cc8beeb04.0000000000000033:head v 32'120, mlcod=29'116)", 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.509841+0000", 2024-05-06T00:27:03.965 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.3496562230000002, 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046528209999999997, 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509841+0000", 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509841+0000", 2024-05-06T00:27:03.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509842+0000", 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0210000000000001e-06 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509851+0000", 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.969e-06 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509866+0000", 2024-05-06T00:27:03.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4915e-05 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509875+0000", 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3449999999999995e-06 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509955+0000", 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0179999999999995e-05 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:03.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510016+0000", 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0207999999999997e-05 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.514458+0000", 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044422430000000002 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.514494+0000", 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5939999999999998e-05 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:03.970 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:03.970 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:03.971 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:03.969+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:03.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:737 write 0xa7a4000 thru 0xa7ab402 (0x7403 bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:738 read 0x35ae800 thru 0x35b7cb0 (0x94b1 bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:739 read 0x80d5c00 thru 0x80df374 (0x9775 bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:741 write 0x8982000 thru 0x89901ec (0xe1ed bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:743 punch from 0x1bfe800 to 0x1c04600, (0x5e00 bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:744 read 0x8927e00 thru 0x893280a (0xaa0b bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:745 write 0x3c89600 thru 0x3c9554a (0xbf4b bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:748 read 0xa762400 thru 0xa76f89f (0xd4a0 bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:749 write 0x90a1800 thru 0x90aba08 (0xa209 bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:754 write 0xdf40000 thru 0xdf4d2a4 (0xd2a5 bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:756 trunc from 0xdf4d400 to 0xba67800 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:757 punch from 0x97fa400 to 0x9802600, (0x8200 bytes) 2024-05-06T00:27:03.974 INFO:teuthology.orchestra.run.smithi186.stdout:758 write 0x739bc00 thru 0x739be5b (0x25c bytes) 2024-05-06T00:27:03.975 INFO:teuthology.orchestra.run.smithi186.stdout:760 read 0x6837600 thru 0x68438af (0xc2b0 bytes) 2024-05-06T00:27:03.975 INFO:teuthology.orchestra.run.smithi186.stdout:766 punch from 0x6c7e200 to 0x6c8da00, (0xf800 bytes) 2024-05-06T00:27:03.975 INFO:teuthology.orchestra.run.smithi186.stdout:767 trunc from 0xba67800 to 0x1059c00 2024-05-06T00:27:04.029 INFO:teuthology.orchestra.run.smithi186.stdout:770 write 0x769fe00 thru 0x76a0eda (0x10db bytes) 2024-05-06T00:27:04.031 INFO:teuthology.orchestra.run.smithi186.stdout:771 read 0x1300600 thru 0x130a0dc (0x9add bytes) 2024-05-06T00:27:04.032 INFO:teuthology.orchestra.run.smithi186.stdout:772 write 0x1fb1a00 thru 0x1fb1ac7 (0xc8 bytes) 2024-05-06T00:27:04.038 INFO:teuthology.orchestra.run.smithi186.stdout:774 trunc from 0x76a1000 to 0x4e8ba00 2024-05-06T00:27:04.058 INFO:teuthology.orchestra.run.smithi186.stdout:775 punch from 0x3ba5200 to 0x3baea00, (0x9800 bytes) 2024-05-06T00:27:04.072 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:04.073+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:04.097 INFO:teuthology.orchestra.run.smithi186.stdout:777 write 0xcf54600 thru 0xcf55c14 (0x1615 bytes) 2024-05-06T00:27:04.100 INFO:teuthology.orchestra.run.smithi186.stdout:778 punch from 0x341de00 to 0x3427000, (0x9200 bytes) 2024-05-06T00:27:04.121 INFO:teuthology.orchestra.run.smithi186.stdout:779 write 0xd2d3400 thru 0xd2d46ce (0x12cf bytes) 2024-05-06T00:27:04.154 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:04.155 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:04.155 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:04.155 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:04.173 INFO:teuthology.orchestra.run.smithi186.stdout:780 trunc from 0xd2d4800 to 0x1a3000 2024-05-06T00:27:04.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:04.175 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:04.173+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:04.282 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:04.281+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:04.286 INFO:teuthology.orchestra.run.smithi186.stdout:781 read 0x110c00 thru 0x1177e3 (0x6be4 bytes) 2024-05-06T00:27:04.289 INFO:teuthology.orchestra.run.smithi186.stdout:783 read 0x16b600 thru 0x172d9c (0x779d bytes) 2024-05-06T00:27:04.291 INFO:teuthology.orchestra.run.smithi186.stdout:784 read 0xe7000 thru 0xeaf1e (0x3f1f bytes) 2024-05-06T00:27:04.293 INFO:teuthology.orchestra.run.smithi186.stdout:785 read 0x104e00 thru 0x10b365 (0x6566 bytes) 2024-05-06T00:27:04.295 INFO:teuthology.orchestra.run.smithi186.stdout:786 read 0x16a200 thru 0x1789d5 (0xe7d6 bytes) 2024-05-06T00:27:04.296 INFO:teuthology.orchestra.run.smithi186.stdout:787 clone 6 order 22 su 131072 sc 13 2024-05-06T00:27:04.355 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:04.355 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:04.355 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:04.355 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:04.356 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:04.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:04.379 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:04.377+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:04.479 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:04.477+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:10 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.776646879000001, 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074329959999999999, 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 10 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725263+0000", 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725266+0000", 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8830000000000002e-06 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725276+0000", 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.042e-05 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725297+0000", 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0927000000000001e-05 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725314+0000", 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7266e-05 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725442+0000", 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012823899999999999 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.725452+0000", 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.944e-06 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729257+0000", 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038050530000000001 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729483+0000", 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022601199999999999 2024-05-06T00:27:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.729822+0000", 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033856400000000002 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.731316+0000", 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014941209999999999 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732407+0000", 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010905820000000001 2024-05-06T00:27:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732553+0000", 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014660899999999999 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.732696+0000", 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014237599999999999 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:556 3.17 e23/18 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head v 23'22, mlcod=23'20)", 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.814744919000001, 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039476930000000004, 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687165+0000", 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687167+0000", 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9269999999999999e-06 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687457+0000", 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029068099999999999 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687495+0000", 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7870000000000002e-05 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687514+0000", 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8419000000000001e-05 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687650+0000", 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136966 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687692+0000", 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1029999999999998e-05 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690995+0000", 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033034140000000002 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691112+0000", 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117386 2024-05-06T00:27:04.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:557 3.17 e23/18 3:eac2123b:::rbd_data.110c17d674e9.0000000000000005:head v 23'23, mlcod=23'20)", 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.814343632, 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039228129999999998, 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687566+0000", 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687567+0000", 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2869999999999999e-06 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687619+0000", 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2119e-05 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687640+0000", 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1123000000000001e-05 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.687656+0000", 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.523e-05 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688969+0000", 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00131369 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.689027+0000", 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000001e-05 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691373+0000", 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023457819999999998 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691489+0000", 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116042 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4412.0:558 3.17 e23/18 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head v 23'24, mlcod=23'20)", 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.813686597, 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037091110000000002, 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.519 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688223+0000", 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688225+0000", 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4940000000000002e-06 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688325+0000", 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9679000000000006e-05 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688356+0000", 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1313000000000003e-05 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.688377+0000", 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0970000000000001e-05 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690069+0000", 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016917360000000001 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.690106+0000", 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6912999999999997e-05 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691873+0000", 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017671200000000001 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.691932+0000", 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8885999999999998e-05 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:558 3.e 3:72699893:::rbd_header.110cdb7de3c5:head [watch watch cookie 139651801975840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-05-06T00:27:04.523 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.159824567999999, 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044008279999999999, 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:04.524 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 558 2024-05-06T00:27:04.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.525 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:27:04.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342085+0000", 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342087+0000", 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8560000000000001e-06 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342095+0000", 2024-05-06T00:27:04.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342119+0000", 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3632999999999999e-05 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342142+0000", 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3459999999999999e-05 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342227+0000", 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4715999999999993e-05 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342358+0000", 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013112700000000001 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.342590+0000", 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023153400000000001 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.343979+0000", 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013895909999999999 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346305+0000", 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002325321 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346378+0000", 2024-05-06T00:27:04.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3276999999999997e-05 2024-05-06T00:27:04.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:51.346486+0000", 2024-05-06T00:27:04.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010778299999999999 2024-05-06T00:27:04.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.530 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:1046 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:54.887078+0000", 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.6148320219999999, 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033926130000000001, 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1046 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.887078+0000", 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.887078+0000", 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.887080+0000", 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3130000000000001e-06 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.887091+0000", 2024-05-06T00:27:04.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1165999999999999e-05 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.887122+0000", 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0889999999999997e-05 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.887137+0000", 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4924e-05 2024-05-06T00:27:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.887290+0000", 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015358200000000001 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.887531+0000", 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000240685 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.888139+0000", 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00060807500000000002 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.889525+0000", 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001386242 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.890326+0000", 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080049099999999996 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.890408+0000", 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1613000000000006e-05 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:54.890470+0000", 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2632000000000006e-05 2024-05-06T00:27:04.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:133 3.f 3:f05fae5f:::rbd_data.110cc8beeb04.0000000000000037:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3604480~14848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.654212+0000", 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.8476977209999994, 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039958529999999997, 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 133 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.654212+0000", 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.654212+0000", 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.654216+0000", 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8800000000000001e-06 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.654297+0000", 2024-05-06T00:27:04.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1600000000000005e-05 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.654339+0000", 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1752999999999998e-05 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.654363+0000", 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3569000000000001e-05 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.654557+0000", 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019457200000000001 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.654797+0000", 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024027200000000001 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.655186+0000", 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038864299999999999 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.657045+0000", 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018591759999999999 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.657916+0000", 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087038299999999996 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.658076+0000", 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016044200000000001 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.658208+0000", 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013156299999999999 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:169 3.1d 3:b8eceabe:::rbd_data.110cc8beeb04.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3436544~49152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.690766+0000", 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.8111435700000005, 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045818220000000002, 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.544 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 169 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.690766+0000", 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.690766+0000", 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.690768+0000", 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3800000000000001e-06 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.690935+0000", 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016623199999999999 2024-05-06T00:27:04.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.690978+0000", 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3683000000000002e-05 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.691001+0000", 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2722999999999999e-05 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.691181+0000", 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017955900000000001 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.691546+0000", 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036548400000000001 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.691947+0000", 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040046999999999999 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.693914+0000", 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019676720000000002 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.695184+0000", 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270263 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.695269+0000", 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4079999999999995e-05 2024-05-06T00:27:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.695348+0000", 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9276000000000005e-05 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:301 3.5 3:a63c39f3:::rbd_data.110cc8beeb04.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7332387279999999, 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010279264999999999, 2024-05-06T00:27:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 301 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768675+0000", 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768804+0000", 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012949899999999999 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768847+0000", 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2953e-05 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768867+0000", 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9859000000000001e-05 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.769041+0000", 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017415900000000001 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.769059+0000", 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.774e-05 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774608+0000", 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055493640000000002 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774935+0000", 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000326464 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.775337+0000", 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040168099999999998 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.777983+0000", 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026467119999999998 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778711+0000", 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00072804800000000002 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778876+0000", 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164869 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778950+0000", 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3873000000000002e-05 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.556 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:302 3.c 3:3742b31b:::rbd_data.110cc8beeb04.000000000000001c:head [set-alloc-hint object_size 4194304 write_size 4194304,write 274432~12288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.768908+0000", 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7330014459999994, 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082404130000000003, 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 302 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768908+0000", 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768908+0000", 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768910+0000", 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8330000000000001e-06 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768940+0000", 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9723e-05 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768961+0000", 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1736999999999999e-05 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768979+0000", 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7575999999999999e-05 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.769128+0000", 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014886599999999999 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.769160+0000", 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2398999999999999e-05 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774444+0000", 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052842399999999999 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774695+0000", 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025064500000000002 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774897+0000", 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000201702 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776704+0000", 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001807672 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776902+0000", 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019720600000000001 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.777081+0000", 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017971799999999999 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.777149+0000", 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7095999999999996e-05 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:319 3.1d 3:b8eceabe:::rbd_data.110cc8beeb04.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.792614+0000", 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7092956360000002, 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062439219999999998, 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 319 2024-05-06T00:27:04.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792614+0000", 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792614+0000", 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792617+0000", 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7939999999999998e-06 2024-05-06T00:27:04.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792627+0000", 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0506000000000001e-05 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792660+0000", 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3213000000000002e-05 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792681+0000", 2024-05-06T00:27:04.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0780000000000001e-05 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792847+0000", 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016599900000000001 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793116+0000", 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026865800000000002 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:04.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793390+0000", 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027396099999999997 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795063+0000", 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016732369999999999 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798620+0000", 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035567329999999999 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798758+0000", 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807199999999999 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798858+0000", 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9969000000000005e-05 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:321 3.5 3:a63c39f3:::rbd_data.110cc8beeb04.000000000000000f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.792723+0000", 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7091864300000008, 2024-05-06T00:27:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051720200000000003, 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 321 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792723+0000", 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792723+0000", 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792725+0000", 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.474e-06 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792735+0000", 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9960000000000005e-06 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792759+0000", 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3915999999999999e-05 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792773+0000", 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4857e-05 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792971+0000", 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019796500000000001 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793180+0000", 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000208669 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793456+0000", 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027571799999999999 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794968+0000", 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015120909999999999 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797656+0000", 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026877789999999999 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797788+0000", 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013223600000000001 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797895+0000", 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010731899999999999 2024-05-06T00:27:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:1206 3.5 3:a63c39f3:::rbd_data.110cc8beeb04.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2840064~5632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:56.382233+0000", 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.1196762289999995, 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045019279999999997, 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1206 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382233+0000", 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382233+0000", 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382236+0000", 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6860000000000002e-06 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382254+0000", 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8430000000000001e-05 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382279+0000", 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4686999999999999e-05 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382314+0000", 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4406000000000002e-05 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382434+0000", 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012093499999999999 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382630+0000", 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019526600000000001 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.382956+0000", 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032671700000000002 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.384382+0000", 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014255520000000001 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.386496+0000", 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002114168 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.386623+0000", 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127329 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.386735+0000", 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111752 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:1208 3.1d 3:bdf82505:::rbd_data.110cc8beeb04.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 4023808~39424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:56.383722+0000", 2024-05-06T00:27:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.1181873279999994, 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044853050000000002, 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1208 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.383722+0000", 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.383722+0000", 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.383725+0000", 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4930000000000001e-06 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.383833+0000", 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010864 2024-05-06T00:27:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.383857+0000", 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3609e-05 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.383870+0000", 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3257e-05 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.384021+0000", 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150679 2024-05-06T00:27:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.384425+0000", 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040429000000000002 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.384649+0000", 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022372200000000001 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.387040+0000", 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023911179999999998 2024-05-06T00:27:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.387974+0000", 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093419200000000003 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.388112+0000", 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013737600000000001 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:56.388208+0000", 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5928999999999996e-05 2024-05-06T00:27:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2529 3.f 3:f145a3bb:::rbd_object_map.110cc8beeb04.0000000000000006:head [writefull 0~50] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.324995+0000", 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.176914419, 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040525500000000002, 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2529 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.324995+0000", 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.324995+0000", 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.324998+0000", 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8279999999999999e-06 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.325009+0000", 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0641e-05 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.325040+0000", 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.146e-05 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.325052+0000", 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1681e-05 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.325137+0000", 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5504e-05 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.325144+0000", 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.238e-06 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.327293+0000", 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021488380000000001 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.327436+0000", 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014254400000000001 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.327597+0000", 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016085000000000001 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.328365+0000", 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076864800000000003 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.328968+0000", 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00060297499999999995 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.329015+0000", 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6675000000000003e-05 2024-05-06T00:27:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.329048+0000", 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2668e-05 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2575 3.1d 3:b8eceabe:::rbd_data.110cc8beeb04.000000000000000e:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.504420+0000", 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.9974890910000003, 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053221020000000004, 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2575 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504420+0000", 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504420+0000", 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504423+0000", 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6139999999999999e-06 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504433+0000", 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8970000000000003e-06 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504460+0000", 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7146999999999999e-05 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1646e-05 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504601+0000", 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012944899999999999 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504697+0000", 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6179000000000002e-05 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504907+0000", 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020989000000000001 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.506130+0000", 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001222934 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509612+0000", 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034813869999999998 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509683+0000", 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0951000000000006e-05 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509743+0000", 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0007999999999999e-05 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2579 3.16 3:68d1dfa3:::rbd_data.110cc8beeb04.000000000000002a:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.504503+0000", 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.9974067020000001, 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041206070000000001, 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2579 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504503+0000", 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504503+0000", 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504504+0000", 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5899999999999995e-07 2024-05-06T00:27:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504508+0000", 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6580000000000001e-06 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504522+0000", 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3249e-05 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504530+0000", 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1440000000000003e-06 2024-05-06T00:27:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504597+0000", 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7275999999999998e-05 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504679+0000", 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1711000000000004e-05 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504851+0000", 2024-05-06T00:27:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017227900000000001 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.506121+0000", 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012696140000000001 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508537+0000", 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024159749999999999 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508582+0000", 2024-05-06T00:27:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5983999999999999e-05 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508623+0000", 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0958000000000003e-05 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2589 3.16 3:68dfe539:::rbd_data.110cc8beeb04.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.508849+0000", 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.993060179, 2024-05-06T00:27:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0058650179999999996, 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2589 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508849+0000", 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508849+0000", 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508851+0000", 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2130000000000001e-06 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508857+0000", 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3879999999999997e-06 2024-05-06T00:27:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508876+0000", 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9040000000000001e-05 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508888+0000", 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1596e-05 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.508969+0000", 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0945999999999994e-05 2024-05-06T00:27:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509035+0000", 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6778000000000003e-05 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.509240+0000", 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205136 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.511749+0000", 2024-05-06T00:27:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025083200000000001 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.514569+0000", 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028197610000000001 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.514652+0000", 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3861000000000002e-05 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.514714+0000", 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1978999999999997e-05 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4528.0:1805 3.1c 3:3f657bd3:::rbd_data.110c7776112a.0000000000000000:head [read 69632~61440] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:03.968955+0000", 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.53295415800000001, 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033684460000000002, 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4528", 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1805 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968955+0000", 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968955+0000", 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968956+0000", 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7000000000000004e-07 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968960+0000", 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0489999999999997e-06 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968970+0000", 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0199e-05 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968979+0000", 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5590000000000006e-06 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.970380+0000", 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00140145 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.970469+0000", 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8777999999999996e-05 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.972053+0000", 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00158359 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.972162+0000", 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109613 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.972324+0000", 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016143799999999999 2024-05-06T00:27:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4528.0:1806 3.1c 3:3f657bd3:::rbd_data.110c7776112a.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:03.968989+0000", 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.53292009399999996, 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037662279999999999, 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4528", 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1806 2024-05-06T00:27:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968989+0000", 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968989+0000", 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968990+0000", 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5000000000000002e-07 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.968993+0000", 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2839999999999998e-06 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.969003+0000", 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6309999999999998e-06 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.969011+0000", 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8050000000000007e-06 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.970511+0000", 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001499809 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.970583+0000", 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2502000000000005e-05 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.972382+0000", 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017993219999999999 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.972536+0000", 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015387199999999999 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:03.972756+0000", 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021935300000000001 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:04.618 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:04.619 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:04.620 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:04.617+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:04.720 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:04.717+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:04.754 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:04.755 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:04.755 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:04.768 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:27:04.841 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:04.837+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:04.922 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:04.921+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:04.936 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:04.936 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:04.936 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:04.937 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:04.937 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:04.952 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:05.023 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:05.021+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:05.027 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone5 from 0x1a3000 (overlap 0x1a3000) to 0xa7800 2024-05-06T00:27:05.075 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:558 3.17 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write 5419008~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.387613358999999, 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065722799999999998, 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 558 2024-05-06T00:27:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685115+0000", 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9399999999999995e-07 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685137+0000", 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2628000000000001e-05 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685145+0000", 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6310000000000003e-06 2024-05-06T00:27:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685152+0000", 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3889999999999999e-06 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686121+0000", 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096883500000000003 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686199+0000", 2024-05-06T00:27:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7911000000000007e-05 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686364+0000", 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000164754 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.690386+0000", 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040223309999999996 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691639+0000", 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012525640000000001 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691667+0000", 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8084000000000001e-05 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691686+0000", 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9658999999999999e-05 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:689 3.6 3:64805dc9:::rbd_data.110c17d674e9.000000000000000e:head [set-alloc-hint object_size 16777216 write_size 16777216,write 3723264~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.071527908, 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611419999999999, 2024-05-06T00:27:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 689 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001201+0000", 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3859999999999999e-06 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001252+0000", 2024-05-06T00:27:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1165999999999998e-05 2024-05-06T00:27:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001270+0000", 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7813e-05 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001281+0000", 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1209999999999999e-05 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001382+0000", 2024-05-06T00:27:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010090300000000001 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001599+0000", 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021734900000000001 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001815+0000", 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021530799999999999 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.003840+0000", 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020251010000000001 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007312+0000", 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034724130000000001 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007390+0000", 2024-05-06T00:27:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7810000000000002e-05 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007461+0000", 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0683000000000001e-05 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.2683682139999997, 2024-05-06T00:27:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:27:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:27:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.2683520110000002, 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:27:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2527 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch ping cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9122580620000003, 2024-05-06T00:27:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16552214100000001, 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2527 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160471+0000", 2024-05-06T00:27:05.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.206e-06 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160477+0000", 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7440000000000001e-06 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160491+0000", 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3739e-05 2024-05-06T00:27:05.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160506+0000", 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.473e-05 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160712+0000", 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020643899999999999 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160736+0000", 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3676000000000002e-05 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325517+0000", 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16478073700000001 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325685+0000", 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016879300000000001 2024-05-06T00:27:05.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325992+0000", 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030607699999999998 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2558 3.1f 3:fcc12fb6:::rbd_data.110cc8beeb04.0000000000000000:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5743359659999996, 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067308660000000003, 2024-05-06T00:27:05.109 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2558 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498394+0000", 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2469999999999998e-06 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498404+0000", 2024-05-06T00:27:05.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0577000000000001e-05 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498435+0000", 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0743e-05 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498452+0000", 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6650000000000002e-05 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498620+0000", 2024-05-06T00:27:05.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168202 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498840+0000", 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022057199999999999 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.499360+0000", 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000519149 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:05.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502603+0000", 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032430419999999998 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504911+0000", 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023087350000000001 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505025+0000", 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011343 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505122+0000", 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7518999999999999e-05 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1858 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5718358779999999, 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069279169999999996, 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1858 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500894+0000", 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.587e-06 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500907+0000", 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.29e-05 2024-05-06T00:27:05.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500940+0000", 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2709999999999997e-05 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500961+0000", 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0916000000000001e-05 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507119+0000", 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061583690000000003 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507405+0000", 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028573800000000001 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507820+0000", 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000414697 2024-05-06T00:27:05.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2571 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5714584240000002, 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079485920000000009, 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2571 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501271+0000", 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3700000000000002e-06 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501290+0000", 2024-05-06T00:27:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8505999999999999e-05 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501321+0000", 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.099e-05 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501340+0000", 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8987e-05 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508595+0000", 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072554910000000002 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508787+0000", 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019116399999999999 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509218+0000", 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043108399999999998 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1859 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.570646784, 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078054739999999997, 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1859 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502083+0000", 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7070000000000001e-06 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502097+0000", 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3447e-05 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502130+0000", 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3300000000000003e-05 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502151+0000", 2024-05-06T00:27:05.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0665999999999998e-05 2024-05-06T00:27:05.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509314+0000", 2024-05-06T00:27:05.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071635980000000002 2024-05-06T00:27:05.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509492+0000", 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000177975 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509886+0000", 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000393781 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2572 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5705682200000002, 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085800230000000009, 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2572 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502162+0000", 2024-05-06T00:27:05.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.993e-06 2024-05-06T00:27:05.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502177+0000", 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.506e-05 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502206+0000", 2024-05-06T00:27:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8379000000000001e-05 2024-05-06T00:27:05.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502226+0000", 2024-05-06T00:27:05.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9927000000000001e-05 2024-05-06T00:27:05.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510022+0000", 2024-05-06T00:27:05.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077960959999999998 2024-05-06T00:27:05.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510238+0000", 2024-05-06T00:27:05.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021639200000000001 2024-05-06T00:27:05.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510739+0000", 2024-05-06T00:27:05.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00050117599999999999 2024-05-06T00:27:05.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.144 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.145 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2573 3.12 3:4926f074:::rbd_data.110cc8beeb04.0000000000000006:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.145 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:05.145 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5699897920000003, 2024-05-06T00:27:05.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066001020000000001, 2024-05-06T00:27:05.146 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:05.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2573 2024-05-06T00:27:05.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.147 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:05.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:05.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502740+0000", 2024-05-06T00:27:05.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.88e-06 2024-05-06T00:27:05.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502749+0000", 2024-05-06T00:27:05.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8400000000000007e-06 2024-05-06T00:27:05.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502771+0000", 2024-05-06T00:27:05.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1622999999999999e-05 2024-05-06T00:27:05.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502788+0000", 2024-05-06T00:27:05.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7e-05 2024-05-06T00:27:05.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502947+0000", 2024-05-06T00:27:05.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015869300000000001 2024-05-06T00:27:05.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503160+0000", 2024-05-06T00:27:05.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000213291 2024-05-06T00:27:05.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:05.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503643+0000", 2024-05-06T00:27:05.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048288900000000002 2024-05-06T00:27:05.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:05.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507989+0000", 2024-05-06T00:27:05.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043459170000000004 2024-05-06T00:27:05.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:05.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509041+0000", 2024-05-06T00:27:05.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010519209999999999 2024-05-06T00:27:05.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:05.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509218+0000", 2024-05-06T00:27:05.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017686899999999999 2024-05-06T00:27:05.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509338+0000", 2024-05-06T00:27:05.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120179 2024-05-06T00:27:05.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.159 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2580 3.12 3:4e19e050:::rbd_data.110cc8beeb04.000000000000002d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.159 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:05.159 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5698748389999997, 2024-05-06T00:27:05.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079317220000000004, 2024-05-06T00:27:05.160 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:05.160 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2580 2024-05-06T00:27:05.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.161 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:05.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:05.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502854+0000", 2024-05-06T00:27:05.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:05.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502862+0000", 2024-05-06T00:27:05.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9140000000000008e-06 2024-05-06T00:27:05.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502882+0000", 2024-05-06T00:27:05.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9885999999999999e-05 2024-05-06T00:27:05.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502896+0000", 2024-05-06T00:27:05.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4219999999999999e-05 2024-05-06T00:27:05.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505092+0000", 2024-05-06T00:27:05.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021958530000000002 2024-05-06T00:27:05.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505289+0000", 2024-05-06T00:27:05.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019717899999999999 2024-05-06T00:27:05.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505714+0000", 2024-05-06T00:27:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042437900000000001 2024-05-06T00:27:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508626+0000", 2024-05-06T00:27:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029123479999999999 2024-05-06T00:27:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510454+0000", 2024-05-06T00:27:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001828292 2024-05-06T00:27:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510664+0000", 2024-05-06T00:27:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000209829 2024-05-06T00:27:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510784+0000", 2024-05-06T00:27:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120386 2024-05-06T00:27:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2581 3.1b 3:d89117cf:::rbd_data.110cc8beeb04.0000000000000033:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.569804456, 2024-05-06T00:27:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010296013, 2024-05-06T00:27:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2581 2024-05-06T00:27:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502924+0000", 2024-05-06T00:27:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.243e-06 2024-05-06T00:27:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502931+0000", 2024-05-06T00:27:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-06T00:27:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502948+0000", 2024-05-06T00:27:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7374e-05 2024-05-06T00:27:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502962+0000", 2024-05-06T00:27:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3653e-05 2024-05-06T00:27:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503093+0000", 2024-05-06T00:27:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013094499999999999 2024-05-06T00:27:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503278+0000", 2024-05-06T00:27:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000185582 2024-05-06T00:27:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503709+0000", 2024-05-06T00:27:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043043999999999998 2024-05-06T00:27:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507476+0000", 2024-05-06T00:27:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003766901 2024-05-06T00:27:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513089+0000", 2024-05-06T00:27:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005613743 2024-05-06T00:27:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513164+0000", 2024-05-06T00:27:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4276999999999994e-05 2024-05-06T00:27:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513219+0000", 2024-05-06T00:27:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5432000000000001e-05 2024-05-06T00:27:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1860 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5682586919999997, 2024-05-06T00:27:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096507989999999998, 2024-05-06T00:27:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1860 2024-05-06T00:27:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.027e-06 2024-05-06T00:27:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504488+0000", 2024-05-06T00:27:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6629999999999998e-05 2024-05-06T00:27:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504523+0000", 2024-05-06T00:27:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4178999999999999e-05 2024-05-06T00:27:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504544+0000", 2024-05-06T00:27:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1627000000000001e-05 2024-05-06T00:27:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513785+0000", 2024-05-06T00:27:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092403420000000003 2024-05-06T00:27:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513895+0000", 2024-05-06T00:27:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110796 2024-05-06T00:27:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514120+0000", 2024-05-06T00:27:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000224198 2024-05-06T00:27:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2585 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051770013168] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.505254+0000", 2024-05-06T00:27:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.567473734, 2024-05-06T00:27:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064207559999999997, 2024-05-06T00:27:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2585 2024-05-06T00:27:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505254+0000", 2024-05-06T00:27:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505254+0000", 2024-05-06T00:27:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505255+0000", 2024-05-06T00:27:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505264+0000", 2024-05-06T00:27:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5399999999999996e-06 2024-05-06T00:27:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505284+0000", 2024-05-06T00:27:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0237e-05 2024-05-06T00:27:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505300+0000", 2024-05-06T00:27:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5549e-05 2024-05-06T00:27:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511267+0000", 2024-05-06T00:27:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005967717 2024-05-06T00:27:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511381+0000", 2024-05-06T00:27:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114169 2024-05-06T00:27:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511674+0000", 2024-05-06T00:27:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029310799999999999 2024-05-06T00:27:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2586 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051770034032] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5674056350000001, 2024-05-06T00:27:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070106719999999999, 2024-05-06T00:27:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2586 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505323+0000", 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.243e-06 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505330+0000", 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1169999999999998e-06 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505349+0000", 2024-05-06T00:27:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8482999999999999e-05 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505368+0000", 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9576999999999999e-05 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511761+0000", 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063923230000000001 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511904+0000", 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014312600000000001 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.512333+0000", 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042880300000000001 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2588 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.507028+0000", 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5656990220000004, 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062123020000000003, 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2588 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507028+0000", 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507028+0000", 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507030+0000", 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.587e-06 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507041+0000", 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.101e-05 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507066+0000", 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5126999999999998e-05 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507084+0000", 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8278999999999999e-05 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.512806+0000", 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057212160000000003 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.512967+0000", 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000160965 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513241+0000", 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027411800000000001 2024-05-06T00:27:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2590 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051770011072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.507509+0000", 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5652188020000004, 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061961129999999996, 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2590 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507509+0000", 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507509+0000", 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507511+0000", 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1500000000000002e-06 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507539+0000", 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7742999999999998e-05 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507570+0000", 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1619999999999999e-05 2024-05-06T00:27:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507590+0000", 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9919999999999999e-05 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513312+0000", 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057217659999999997 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513435+0000", 2024-05-06T00:27:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123499 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513705+0000", 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026941500000000001 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1862 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5641644000000001, 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006446907, 2024-05-06T00:27:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1862 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508565+0000", 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508574+0000", 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2199999999999998e-06 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508595+0000", 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0783000000000001e-05 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508610+0000", 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5367e-05 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514622+0000", 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060122470000000001 2024-05-06T00:27:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514725+0000", 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103009 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515010+0000", 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028462099999999998 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2591 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051769985984] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5637566469999999, 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066352190000000004, 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2591 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508973+0000", 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.39e-06 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508983+0000", 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0254e-05 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509010+0000", 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6398999999999999e-05 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509033+0000", 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.298e-05 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515092+0000", 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060590740000000002 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515201+0000", 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108602 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515606+0000", 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040551999999999998 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:05.232 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:05.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:05.234 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:05.233+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:05.334 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:05.333+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:05.364 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:05.365 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:05.378 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:05.435 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:05.433+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:05.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:05.498 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:05.498 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:05.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:05.536 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:05.537+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:05.580 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:05.580 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:05.596 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:05.597 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:05.628 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:05.629 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:05.637 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:05.633+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:05.642 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:05.642 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:05.644 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-05-06T00:27:05.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:05.652 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:05.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:05.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:05.675 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:05.696 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:05.758 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:05.757+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:05.795 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:05.795 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:05.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:05.854 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:05.853+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:05.940 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:05.940+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:05.984 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:05.985 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:06.001 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:06.049 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:06.049+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:06.128 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:06.128 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:06.141 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:06.141+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:06.143 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:06.226 INFO:teuthology.orchestra.run.smithi186.stdout:788 read 0x16400 thru 0x1ee27 (0x8a28 bytes) 2024-05-06T00:27:06.243 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:06.240+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:06.270 INFO:teuthology.orchestra.run.smithi186.stdout:789 trunc from 0x1a3000 to 0x7b57e00 2024-05-06T00:27:06.275 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:06.275 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:06.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:06.301 INFO:teuthology.orchestra.run.smithi186.stdout:791 write 0x4cf6000 thru 0x4cf8434 (0x2435 bytes) 2024-05-06T00:27:06.304 INFO:teuthology.orchestra.run.smithi186.stdout:793 punch from 0x1bd5000 to 0x1bdfa00, (0xaa00 bytes) 2024-05-06T00:27:06.343 INFO:teuthology.orchestra.run.smithi186.stdout:794 write 0xa197c00 thru 0xa19fbea (0x7feb bytes) 2024-05-06T00:27:06.345 INFO:teuthology.orchestra.run.smithi186.stdout:797 write 0x1016800 thru 0x101d609 (0x6e0a bytes) 2024-05-06T00:27:06.345 INFO:teuthology.orchestra.run.smithi186.stdout:799 write 0x50d3600 thru 0x50dff60 (0xc961 bytes) 2024-05-06T00:27:06.345 INFO:teuthology.orchestra.run.smithi186.stdout:801 write 0x7bf9800 thru 0x7bf9c61 (0x462 bytes) 2024-05-06T00:27:06.345 INFO:teuthology.orchestra.run.smithi186.stdout:802 read 0x8240600 thru 0x8247bbd (0x75be bytes) 2024-05-06T00:27:06.346 INFO:teuthology.orchestra.run.smithi186.stdout:804 trunc from 0xa19fc00 to 0xa40bc00 2024-05-06T00:27:06.376 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:06.376+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:06.381 INFO:teuthology.orchestra.run.smithi186.stdout:805 write 0x17a4000 thru 0x17a89f1 (0x49f2 bytes) 2024-05-06T00:27:06.383 INFO:teuthology.orchestra.run.smithi186.stdout:806 punch from 0x8db0600 to 0x8db1600, (0x1000 bytes) 2024-05-06T00:27:06.383 INFO:teuthology.orchestra.run.smithi186.stdout:808 write 0x3f4e600 thru 0x3f54c84 (0x6685 bytes) 2024-05-06T00:27:06.383 INFO:teuthology.orchestra.run.smithi186.stdout:809 read 0x6a6f200 thru 0x6a7b701 (0xc502 bytes) 2024-05-06T00:27:06.428 INFO:teuthology.orchestra.run.smithi186.stdout:810 write 0xc55fc00 thru 0xc5633e1 (0x37e2 bytes) 2024-05-06T00:27:06.433 INFO:teuthology.orchestra.run.smithi186.stdout:811 flatten 2024-05-06T00:27:06.445 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:06.444+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:06.459 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:06.459 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:06.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:06.477 INFO:teuthology.orchestra.run.smithi186.stdout:813 punch from 0xa4d8000 to 0xa4e4a00, (0xca00 bytes) 2024-05-06T00:27:06.479 INFO:teuthology.orchestra.run.smithi186.stdout:814 punch from 0x22b0000 to 0x22b9e00, (0x9e00 bytes) 2024-05-06T00:27:06.479 INFO:teuthology.orchestra.run.smithi186.stdout:815 read 0x6b2d000 thru 0x6b3b1b2 (0xe1b3 bytes) 2024-05-06T00:27:06.480 INFO:teuthology.orchestra.run.smithi186.stdout:816 write 0xade2a00 thru 0xade921e (0x681f bytes) 2024-05-06T00:27:06.480 INFO:teuthology.orchestra.run.smithi186.stdout:817 write 0x1854800 thru 0x185ed08 (0xa509 bytes) 2024-05-06T00:27:06.480 INFO:teuthology.orchestra.run.smithi186.stdout:819 read 0x4e96000 thru 0x4e9b516 (0x5517 bytes) 2024-05-06T00:27:06.480 INFO:teuthology.orchestra.run.smithi186.stdout:821 write 0x5e26800 thru 0x5e275d7 (0xdd8 bytes) 2024-05-06T00:27:06.483 INFO:teuthology.orchestra.run.smithi186.stdout:822 write 0x2a2400 thru 0x2a5d1e (0x391f bytes) 2024-05-06T00:27:06.527 INFO:teuthology.orchestra.run.smithi186.stdout:823 write 0xd44cc00 thru 0xd4522c3 (0x56c4 bytes) 2024-05-06T00:27:06.529 INFO:teuthology.orchestra.run.smithi186.stdout:824 read 0x9524600 thru 0x952cb3b (0x853c bytes) 2024-05-06T00:27:06.530 INFO:teuthology.orchestra.run.smithi186.stdout:830 read 0xb2dae00 thru 0xb2e8e49 (0xe04a bytes) 2024-05-06T00:27:06.531 INFO:teuthology.orchestra.run.smithi186.stdout:833 write 0x7e02600 thru 0x7e0cde1 (0xa7e2 bytes) 2024-05-06T00:27:06.545 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:06.545+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:06.571 INFO:teuthology.orchestra.run.smithi186.stdout:834 write 0xe0f9200 thru 0xe1005f4 (0x73f5 bytes) 2024-05-06T00:27:06.574 INFO:teuthology.orchestra.run.smithi186.stdout:836 read 0x20bec00 thru 0x20cb010 (0xc411 bytes) 2024-05-06T00:27:06.575 INFO:teuthology.orchestra.run.smithi186.stdout:837 punch from 0x186f400 to 0x1877200, (0x7e00 bytes) 2024-05-06T00:27:06.576 INFO:teuthology.orchestra.run.smithi186.stdout:839 read 0x83fa600 thru 0x8407ba3 (0xd5a4 bytes) 2024-05-06T00:27:06.577 INFO:teuthology.orchestra.run.smithi186.stdout:843 read 0x3142800 thru 0x314376c (0xf6d bytes) 2024-05-06T00:27:06.577 INFO:teuthology.orchestra.run.smithi186.stdout:845 write 0xb1b4400 thru 0xb1bd9bc (0x95bd bytes) 2024-05-06T00:27:06.578 INFO:teuthology.orchestra.run.smithi186.stdout:847 write 0x1f2a000 thru 0x1f3848d (0xe48e bytes) 2024-05-06T00:27:06.604 INFO:teuthology.orchestra.run.smithi186.stdout:850 trunc from 0xe100600 to 0xbb8c00 2024-05-06T00:27:06.642 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:06.643 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:06.646 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:06.645+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:06.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:06.658 INFO:teuthology.orchestra.run.smithi186.stdout:852 read 0x78ac00 thru 0x78ed38 (0x4139 bytes) 2024-05-06T00:27:06.700 INFO:teuthology.orchestra.run.smithi186.stdout:855 write 0x9f53000 thru 0x9f57b8a (0x4b8b bytes) 2024-05-06T00:27:06.701 INFO:teuthology.orchestra.run.smithi186.stdout:856 read 0x2a6000 thru 0x2ae11e (0x811f bytes) 2024-05-06T00:27:06.703 INFO:teuthology.orchestra.run.smithi186.stdout:857 write 0x3499200 thru 0x34a86db (0xf4dc bytes) 2024-05-06T00:27:06.703 INFO:teuthology.orchestra.run.smithi186.stdout:858 read 0x88ea000 thru 0x88f5b7e (0xbb7f bytes) 2024-05-06T00:27:06.703 INFO:teuthology.orchestra.run.smithi186.stdout:859 read 0x6106000 thru 0x610b777 (0x5778 bytes) 2024-05-06T00:27:06.745 INFO:teuthology.orchestra.run.smithi186.stdout:860 write 0xce4cc00 thru 0xce59b11 (0xcf12 bytes) 2024-05-06T00:27:06.747 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:06.745+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:06.763 INFO:teuthology.orchestra.run.smithi186.stdout:863 trunc from 0xce59c00 to 0x5937c00 2024-05-06T00:27:06.781 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:06.781 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:06.796 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:06.808 INFO:teuthology.orchestra.run.smithi186.stdout:864 read 0x4e6d000 thru 0x4e79ca6 (0xcca7 bytes) 2024-05-06T00:27:06.809 INFO:teuthology.orchestra.run.smithi186.stdout:865 trunc from 0x5937c00 to 0x5c61c00 2024-05-06T00:27:06.830 INFO:teuthology.orchestra.run.smithi186.stdout:867 punch from 0x4f54400 to 0x4f61000, (0xcc00 bytes) 2024-05-06T00:27:06.831 INFO:teuthology.orchestra.run.smithi186.stdout:868 read 0x1bb4800 thru 0x1bbaf76 (0x6777 bytes) 2024-05-06T00:27:06.833 INFO:teuthology.orchestra.run.smithi186.stdout:869 write 0xaf4e00 thru 0xb045b3 (0xf7b4 bytes) 2024-05-06T00:27:06.833 INFO:teuthology.orchestra.run.smithi186.stdout:871 punch from 0x55fb400 to 0x5603a00, (0x8600 bytes) 2024-05-06T00:27:06.848 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:06.845+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:06.866 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:27:06.873 INFO:teuthology.orchestra.run.smithi186.stdout:872 write 0xebe0c00 thru 0xebe3627 (0x2a28 bytes) 2024-05-06T00:27:06.874 INFO:teuthology.orchestra.run.smithi186.stdout:873 write 0xbe4e600 thru 0xbe4f30c (0xd0d bytes) 2024-05-06T00:27:06.874 INFO:teuthology.orchestra.run.smithi186.stdout:874 punch from 0x3738200 to 0x3745800, (0xd600 bytes) 2024-05-06T00:27:06.874 INFO:teuthology.orchestra.run.smithi186.stdout:880 punch from 0xd1b0800 to 0xd1bce00, (0xc600 bytes) 2024-05-06T00:27:06.875 INFO:teuthology.orchestra.run.smithi186.stdout:881 read 0xb435800 thru 0xb435e55 (0x656 bytes) 2024-05-06T00:27:06.875 INFO:teuthology.orchestra.run.smithi186.stdout:883 read 0x3e9c400 thru 0x3ea8ad4 (0xc6d5 bytes) 2024-05-06T00:27:06.881 INFO:teuthology.orchestra.run.smithi186.stdout:885 trunc from 0xebe3800 to 0xaa56a00 2024-05-06T00:27:06.928 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:06.928 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:06.929 INFO:teuthology.orchestra.run.smithi186.stdout:886 write 0xbcb1e00 thru 0xbcc10de (0xf2df bytes) 2024-05-06T00:27:06.932 INFO:teuthology.orchestra.run.smithi186.stdout:887 punch from 0x60b000 to 0x60f600, (0x4600 bytes) 2024-05-06T00:27:06.932 INFO:teuthology.orchestra.run.smithi186.stdout:888 punch from 0x1769800 to 0x1776a00, (0xd200 bytes) 2024-05-06T00:27:06.932 INFO:teuthology.orchestra.run.smithi186.stdout:889 read 0x5f85a00 thru 0x5f912c0 (0xb8c1 bytes) 2024-05-06T00:27:06.935 INFO:teuthology.orchestra.run.smithi186.stdout:890 punch from 0x28b2c00 to 0x28b4600, (0x1a00 bytes) 2024-05-06T00:27:06.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:06.952 INFO:teuthology.orchestra.run.smithi186.stdout:894 write 0xe9b6a00 thru 0xe9b76cc (0xccd bytes) 2024-05-06T00:27:06.982 INFO:teuthology.orchestra.run.smithi186.stdout:895 trunc from 0xe9b7800 to 0x2832c00 2024-05-06T00:27:06.992 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:06.992+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:07.015 INFO:teuthology.orchestra.run.smithi186.stdout:898 read 0x1324c00 thru 0x132fd98 (0xb199 bytes) 2024-05-06T00:27:07.016 INFO:teuthology.orchestra.run.smithi186.stdout:901 read 0xc5c000 thru 0xc5e4d0 (0x24d1 bytes) 2024-05-06T00:27:07.017 INFO:teuthology.orchestra.run.smithi186.stdout:905 read 0xb1d800 thru 0xb1f224 (0x1a25 bytes) 2024-05-06T00:27:07.035 INFO:teuthology.orchestra.run.smithi186.stdout:908 trunc from 0x2832c00 to 0xc93c600 2024-05-06T00:27:07.050 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:07.048+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:07.051 INFO:teuthology.orchestra.run.smithi186.stdout:909 trunc from 0xc93c600 to 0xe77ec00 2024-05-06T00:27:07.062 INFO:teuthology.orchestra.run.smithi186.stdout:910 punch from 0x510be00 to 0x5112000, (0x6200 bytes) 2024-05-06T00:27:07.064 INFO:teuthology.orchestra.run.smithi186.stdout:911 read 0x15e8400 thru 0x15f630c (0xdf0d bytes) 2024-05-06T00:27:07.064 INFO:teuthology.orchestra.run.smithi186.stdout:913 write 0x38b6600 thru 0x38bb545 (0x4f46 bytes) 2024-05-06T00:27:07.064 INFO:teuthology.orchestra.run.smithi186.stdout:915 write 0x3569200 thru 0x3578e79 (0xfc7a bytes) 2024-05-06T00:27:07.065 INFO:teuthology.orchestra.run.smithi186.stdout:918 write 0x226a400 thru 0x226c482 (0x2083 bytes) 2024-05-06T00:27:07.066 INFO:teuthology.orchestra.run.smithi186.stdout:919 read 0x1f6c000 thru 0x1f71d51 (0x5d52 bytes) 2024-05-06T00:27:07.066 INFO:teuthology.orchestra.run.smithi186.stdout:920 write 0x5f64800 thru 0x5f6f095 (0xa896 bytes) 2024-05-06T00:27:07.067 INFO:teuthology.orchestra.run.smithi186.stdout:921 read 0xa7e1000 thru 0xa7e6945 (0x5946 bytes) 2024-05-06T00:27:07.067 INFO:teuthology.orchestra.run.smithi186.stdout:922 write 0xd049e00 thru 0xd04d50c (0x370d bytes) 2024-05-06T00:27:07.067 INFO:teuthology.orchestra.run.smithi186.stdout:923 read 0xccb8400 thru 0xccbb5ca (0x31cb bytes) 2024-05-06T00:27:07.071 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:07.071 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:07.074 INFO:teuthology.orchestra.run.smithi186.stdout:925 trunc from 0xe77ec00 to 0xb5e7200 2024-05-06T00:27:07.086 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:07.104 INFO:teuthology.orchestra.run.smithi186.stdout:927 write 0xb347a00 thru 0xb3493e7 (0x19e8 bytes) 2024-05-06T00:27:07.105 INFO:teuthology.orchestra.run.smithi186.stdout:928 read 0x7c8e800 thru 0x7c9d8a7 (0xf0a8 bytes) 2024-05-06T00:27:07.105 INFO:teuthology.orchestra.run.smithi186.stdout:929 write 0x3305400 thru 0x330db38 (0x8739 bytes) 2024-05-06T00:27:07.106 INFO:teuthology.orchestra.run.smithi186.stdout:932 read 0x2db8e00 thru 0x2dc438b (0xb58c bytes) 2024-05-06T00:27:07.108 INFO:teuthology.orchestra.run.smithi186.stdout:934 write 0x9209400 thru 0x9217d9c (0xe99d bytes) 2024-05-06T00:27:07.108 INFO:teuthology.orchestra.run.smithi186.stdout:935 punch from 0xa3b9600 to 0xa3c0c00, (0x7600 bytes) 2024-05-06T00:27:07.108 INFO:teuthology.orchestra.run.smithi186.stdout:936 clone 7 order 20 su 262144 sc 12 2024-05-06T00:27:07.176 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:07.176+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:07.252 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:07.252+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:07.257 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:07.257 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:07.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:07.354 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:07.353+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:07.436 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:07.436 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:07.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:07.454 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:07.453+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:07.555 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:07.553+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:07.574 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:07.574 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:07.589 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:07.656 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:07.653+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:07.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:07.721 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:07.721 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:07.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:07.758 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:07.756+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:07.846 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:27:07.859 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:07.856+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:07.865 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:07.865 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:07.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:07.885 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:07.916 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:07.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:07.916 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:07.916 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:07.916 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:07.917 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:07.917 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:07.931 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:07.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:07.939 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:07.959 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:07.956+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:07.960 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:07.961 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:07.965 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:07.965 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:27:07.965 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-05-06T00:27:07.965 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-06T00:27:07.965 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-06T00:27:07.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-05-06T00:27:07.981 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:07.981 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:08.057 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:558 3.17 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write 5419008~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.37115404, 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065722799999999998, 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 558 2024-05-06T00:27:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685115+0000", 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9399999999999995e-07 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685137+0000", 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2628000000000001e-05 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685145+0000", 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6310000000000003e-06 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685152+0000", 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3889999999999999e-06 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686121+0000", 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096883500000000003 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686199+0000", 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7911000000000007e-05 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686364+0000", 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000164754 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.690386+0000", 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040223309999999996 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691639+0000", 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012525640000000001 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691667+0000", 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8084000000000001e-05 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691686+0000", 2024-05-06T00:27:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9658999999999999e-05 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:689 3.6 3:64805dc9:::rbd_data.110c17d674e9.000000000000000e:head [set-alloc-hint object_size 16777216 write_size 16777216,write 3723264~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.055068589000001, 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611419999999999, 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 689 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001200+0000", 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001201+0000", 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3859999999999999e-06 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001252+0000", 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1165999999999998e-05 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001270+0000", 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7813e-05 2024-05-06T00:27:08.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001281+0000", 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1209999999999999e-05 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001382+0000", 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010090300000000001 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001599+0000", 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021734900000000001 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.001815+0000", 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021530799999999999 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.003840+0000", 2024-05-06T00:27:08.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020251010000000001 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007312+0000", 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034724130000000001 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007390+0000", 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7810000000000002e-05 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:50.007461+0000", 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0683000000000001e-05 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2519088949999997, 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:27:08.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:27:08.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:08.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2518926920000002, 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:27:08.078 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:27:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:27:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:27:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2527 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch ping cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.8957987430000003, 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16552214100000001, 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2527 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160471+0000", 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.206e-06 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160477+0000", 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7440000000000001e-06 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160491+0000", 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3739e-05 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160506+0000", 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.473e-05 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160712+0000", 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020643899999999999 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160736+0000", 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3676000000000002e-05 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325517+0000", 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16478073700000001 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325685+0000", 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016879300000000001 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325992+0000", 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030607699999999998 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2558 3.1f 3:fcc12fb6:::rbd_data.110cc8beeb04.0000000000000000:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5578766469999996, 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067308660000000003, 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2558 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498394+0000", 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2469999999999998e-06 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498404+0000", 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0577000000000001e-05 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498435+0000", 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0743e-05 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498452+0000", 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6650000000000002e-05 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498620+0000", 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168202 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498840+0000", 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022057199999999999 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.499360+0000", 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000519149 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502603+0000", 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032430419999999998 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504911+0000", 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023087350000000001 2024-05-06T00:27:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505025+0000", 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011343 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505122+0000", 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7518999999999999e-05 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1858 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5553765589999999, 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069279169999999996, 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1858 2024-05-06T00:27:08.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500894+0000", 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.587e-06 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500907+0000", 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.29e-05 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500940+0000", 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2709999999999997e-05 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500961+0000", 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0916000000000001e-05 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507119+0000", 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061583690000000003 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507405+0000", 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028573800000000001 2024-05-06T00:27:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507820+0000", 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000414697 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2571 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5549991050000003, 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079485920000000009, 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2571 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501271+0000", 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3700000000000002e-06 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501290+0000", 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8505999999999999e-05 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501321+0000", 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.099e-05 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501340+0000", 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8987e-05 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508595+0000", 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072554910000000002 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508787+0000", 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019116399999999999 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509218+0000", 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043108399999999998 2024-05-06T00:27:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1859 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.554187465, 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078054739999999997, 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1859 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502083+0000", 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7070000000000001e-06 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502097+0000", 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3447e-05 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502130+0000", 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3300000000000003e-05 2024-05-06T00:27:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502151+0000", 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0665999999999998e-05 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509314+0000", 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071635980000000002 2024-05-06T00:27:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509492+0000", 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000177975 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509886+0000", 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000393781 2024-05-06T00:27:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2572 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5541089010000002, 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085800230000000009, 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2572 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502162+0000", 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.993e-06 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502177+0000", 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.506e-05 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502206+0000", 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8379000000000001e-05 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502226+0000", 2024-05-06T00:27:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9927000000000001e-05 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510022+0000", 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077960959999999998 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510238+0000", 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021639200000000001 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510739+0000", 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00050117599999999999 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2573 3.12 3:4926f074:::rbd_data.110cc8beeb04.0000000000000006:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5535304730000004, 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066001020000000001, 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2573 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502740+0000", 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.88e-06 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502749+0000", 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8400000000000007e-06 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502771+0000", 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1622999999999999e-05 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502788+0000", 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7e-05 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502947+0000", 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015869300000000001 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503160+0000", 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000213291 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503643+0000", 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048288900000000002 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507989+0000", 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043459170000000004 2024-05-06T00:27:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509041+0000", 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010519209999999999 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509218+0000", 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017686899999999999 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509338+0000", 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120179 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2580 3.12 3:4e19e050:::rbd_data.110cc8beeb04.000000000000002d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5534155199999997, 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079317220000000004, 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2580 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502854+0000", 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502862+0000", 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9140000000000008e-06 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502882+0000", 2024-05-06T00:27:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9885999999999999e-05 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502896+0000", 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4219999999999999e-05 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505092+0000", 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021958530000000002 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505289+0000", 2024-05-06T00:27:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019717899999999999 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505714+0000", 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042437900000000001 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508626+0000", 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029123479999999999 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510454+0000", 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001828292 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510664+0000", 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000209829 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510784+0000", 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120386 2024-05-06T00:27:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2581 3.1b 3:d89117cf:::rbd_data.110cc8beeb04.0000000000000033:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.553345137, 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010296013, 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2581 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502924+0000", 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.243e-06 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502931+0000", 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502948+0000", 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7374e-05 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502962+0000", 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3653e-05 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503093+0000", 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013094499999999999 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503278+0000", 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000185582 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503709+0000", 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043043999999999998 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507476+0000", 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003766901 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513089+0000", 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005613743 2024-05-06T00:27:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513164+0000", 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4276999999999994e-05 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513219+0000", 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5432000000000001e-05 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1860 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5517993729999997, 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096507989999999998, 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1860 2024-05-06T00:27:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.027e-06 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504488+0000", 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6629999999999998e-05 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504523+0000", 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4178999999999999e-05 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504544+0000", 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1627000000000001e-05 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513785+0000", 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092403420000000003 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513895+0000", 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110796 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514120+0000", 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000224198 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2585 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051770013168] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.505254+0000", 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.551014415, 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064207559999999997, 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2585 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505254+0000", 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505254+0000", 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505255+0000", 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505264+0000", 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5399999999999996e-06 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505284+0000", 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0237e-05 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505300+0000", 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5549e-05 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511267+0000", 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005967717 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511381+0000", 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114169 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511674+0000", 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029310799999999999 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2586 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051770034032] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5509463160000001, 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070106719999999999, 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2586 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505323+0000", 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.243e-06 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505330+0000", 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1169999999999998e-06 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505349+0000", 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8482999999999999e-05 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505368+0000", 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9576999999999999e-05 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511761+0000", 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063923230000000001 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511904+0000", 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014312600000000001 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.512333+0000", 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042880300000000001 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2588 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.507028+0000", 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5492397029999996, 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062123020000000003, 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2588 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507028+0000", 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507028+0000", 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507030+0000", 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.587e-06 2024-05-06T00:27:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507041+0000", 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.101e-05 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507066+0000", 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5126999999999998e-05 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507084+0000", 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8278999999999999e-05 2024-05-06T00:27:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.512806+0000", 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057212160000000003 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.512967+0000", 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000160965 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513241+0000", 2024-05-06T00:27:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027411800000000001 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2590 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051770011072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.507509+0000", 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5487594829999995, 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061961129999999996, 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2590 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507509+0000", 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507509+0000", 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507511+0000", 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1500000000000002e-06 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507539+0000", 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7742999999999998e-05 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507570+0000", 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1619999999999999e-05 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507590+0000", 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9919999999999999e-05 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513312+0000", 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057217659999999997 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513435+0000", 2024-05-06T00:27:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123499 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513705+0000", 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026941500000000001 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1862 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.5477050810000001, 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006446907, 2024-05-06T00:27:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1862 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508565+0000", 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508574+0000", 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2199999999999998e-06 2024-05-06T00:27:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508595+0000", 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0783000000000001e-05 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508610+0000", 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5367e-05 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514622+0000", 2024-05-06T00:27:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060122470000000001 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514725+0000", 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103009 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515010+0000", 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028462099999999998 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2591 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051769985984] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.547297328, 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066352190000000004, 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2591 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508973+0000", 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.39e-06 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508983+0000", 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0254e-05 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509010+0000", 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6398999999999999e-05 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509033+0000", 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.298e-05 2024-05-06T00:27:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515092+0000", 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060590740000000002 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515201+0000", 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108602 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515606+0000", 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040551999999999998 2024-05-06T00:27:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.168 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:08.170 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:08.170 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-06T00:27:08.170 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:08.170 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:08.171 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:08.168+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:08.270 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:08.269+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:08.299 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:08.299 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:08.314 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:08.406 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:08.405+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:08.472 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:08.469+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:08.487 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:08.487 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:08.487 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:08.487 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:08.487 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:08.501 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:08.573 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:08.572+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:08.606 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-05-06T00:27:08.672 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:228 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 397312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.071915140000002, 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072743440000000003, 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 228 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597016+0000", 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.443e-06 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597023+0000", 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8340000000000001e-06 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597044+0000", 2024-05-06T00:27:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0876000000000001e-05 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597055+0000", 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1314e-05 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597176+0000", 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120565 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:27:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597192+0000", 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6337e-05 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603743+0000", 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065506820000000004 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603976+0000", 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000233355 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604289+0000", 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031293800000000002 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:229 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 528384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.071757241, 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075584060000000002, 2024-05-06T00:27:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 229 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597175+0000", 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8929999999999999e-06 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597184+0000", 2024-05-06T00:27:08.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2040000000000003e-06 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597200+0000", 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5642999999999999e-05 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597209+0000", 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4329999999999993e-06 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597301+0000", 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1588999999999993e-05 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597309+0000", 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604360+0000", 2024-05-06T00:27:08.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070513030000000001 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604516+0000", 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015564999999999999 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604731+0000", 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000215267 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:163 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.985302612, 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060279510000000001, 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 163 2024-05-06T00:27:08.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683627+0000", 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683629+0000", 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.28e-06 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683636+0000", 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2729999999999997e-06 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683662+0000", 2024-05-06T00:27:08.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5754000000000001e-05 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.683673+0000", 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0890000000000001e-05 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.686748+0000", 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030753180000000001 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.686848+0000", 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9697000000000004e-05 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687256+0000", 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040807300000000001 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688723+0000", 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014670740000000001 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689566+0000", 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00084270500000000002 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689616+0000", 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0358999999999997e-05 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689655+0000", 2024-05-06T00:27:08.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9527999999999998e-05 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:165 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.984773498999999, 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010266747, 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 165 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684158+0000", 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6929999999999999e-06 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684170+0000", 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1933000000000001e-05 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684198+0000", 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8266999999999999e-05 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684221+0000", 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2736999999999999e-05 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687824+0000", 2024-05-06T00:27:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003602863 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687915+0000", 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0804999999999995e-05 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688264+0000", 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034917100000000001 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693979+0000", 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057148959999999997 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694148+0000", 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168951 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694315+0000", 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000167016 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694423+0000", 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108415 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:173 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.979263015000001, 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058174089999999999, 2024-05-06T00:27:08.696 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 173 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689667+0000", 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689668+0000", 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2669999999999999e-06 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689676+0000", 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8569999999999995e-06 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689696+0000", 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0006000000000001e-05 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.689711+0000", 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-06T00:27:08.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.691939+0000", 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002227559 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.692171+0000", 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000232475 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693111+0000", 2024-05-06T00:27:08.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000939795 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694929+0000", 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001817281 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695267+0000", 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033894600000000002 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695385+0000", 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117683 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.695484+0000", 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9343000000000001e-05 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:298 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.909438848000001, 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089449070000000002, 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 298 2024-05-06T00:27:08.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759493+0000", 2024-05-06T00:27:08.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8029999999999999e-06 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759503+0000", 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4769999999999992e-06 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759525+0000", 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2067999999999999e-05 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759540+0000", 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5296999999999999e-05 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.762625+0000", 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030855399999999999 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.762723+0000", 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7750000000000004e-05 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763171+0000", 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044755199999999999 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766326+0000", 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031554080000000002 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768328+0000", 2024-05-06T00:27:08.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020015530000000001 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768387+0000", 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9283000000000001e-05 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768436+0000", 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9175999999999998e-05 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:300 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.902516545999999, 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077869669999999997, 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 300 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766415+0000", 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.536e-06 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766423+0000", 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2849999999999995e-06 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766435+0000", 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2121e-05 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766443+0000", 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9140000000000008e-06 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766490+0000", 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6517999999999997e-05 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766495+0000", 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769158+0000", 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026631390000000001 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769268+0000", 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010981599999999999 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769603+0000", 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033493500000000002 2024-05-06T00:27:08.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.773467+0000", 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038639920000000001 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774044+0000", 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00057753000000000004 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774139+0000", 2024-05-06T00:27:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5315999999999993e-05 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774201+0000", 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1054000000000002e-05 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:324 3.19 3:99a6ca97:::rbd_data.110cc8beeb04.0000000000000013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:08.715 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.877921110999999, 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062413850000000003, 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 324 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791010+0000", 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0449999999999999e-06 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791015+0000", 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4689999999999996e-06 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791030+0000", 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4892000000000001e-05 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791038+0000", 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8460000000000008e-06 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791106+0000", 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7994e-05 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791190+0000", 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3337000000000005e-05 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791293+0000", 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103757 2024-05-06T00:27:08.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795405+0000", 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041115479999999996 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797183+0000", 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001778097 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797213+0000", 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.004e-05 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797250+0000", 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7360000000000001e-05 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:328 3.19 3:9fbbd60a:::rbd_data.110cc8beeb04.000000000000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:08.721 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.877772799000001, 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063090830000000001, 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 328 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:08.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791158+0000", 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7000000000000003e-07 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791165+0000", 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3220000000000004e-06 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791178+0000", 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2078e-05 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791184+0000", 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791609+0000", 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042469600000000001 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791782+0000", 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017316400000000001 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.792020+0000", 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023781599999999999 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795867+0000", 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038475010000000001 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797389+0000", 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015214339999999999 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797418+0000", 2024-05-06T00:27:08.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8977999999999999e-05 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797466+0000", 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8489999999999998e-05 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1015 3.8 e29/18 3:1391aa96:::rbd_data.110cc8beeb04.0000000000000025:head v 29'6, mlcod=29'4)", 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.414867694, 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075649460000000003, 2024-05-06T00:27:08.727 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254063+0000", 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4099999999999997e-07 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254072+0000", 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6370000000000005e-06 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254084+0000", 2024-05-06T00:27:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1725999999999999e-05 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254091+0000", 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.379e-06 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254155+0000", 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3906999999999996e-05 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254177+0000", 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1526e-05 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261558+0000", 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073814409999999999 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261627+0000", 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9388999999999996e-05 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.731 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1017 3.7 e29/18 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head v 29'11, mlcod=29'9)", 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.414822706000001, 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075152739999999997, 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254108+0000", 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254112+0000", 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254119+0000", 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7800000000000003e-06 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254124+0000", 2024-05-06T00:27:08.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8509999999999998e-06 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254189+0000", 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4757000000000004e-05 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254209+0000", 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0307000000000001e-05 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261566+0000", 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007356596 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261623+0000", 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7111000000000003e-05 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1022 3.8 e29/18 3:17fe3906:::rbd_data.110cc8beeb04.0000000000000036:head v 29'7, mlcod=29'4)", 2024-05-06T00:27:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.41466059, 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073889450000000001, 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254270+0000", 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254275+0000", 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6120000000000001e-06 2024-05-06T00:27:08.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254283+0000", 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2199999999999992e-06 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254289+0000", 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9880000000000001e-06 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254360+0000", 2024-05-06T00:27:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1062000000000005e-05 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254379+0000", 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9194e-05 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261632+0000", 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00725255 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261658+0000", 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6857000000000001e-05 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1018 3.7 e29/18 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head v 29'12, mlcod=29'9)", 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.414608763, 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073213799999999997, 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254322+0000", 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5400000000000002e-07 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254326+0000", 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5979999999999999e-06 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254334+0000", 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1489999999999997e-06 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254339+0000", 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1399999999999999e-06 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254387+0000", 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8102000000000001e-05 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254403+0000", 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5982e-05 2024-05-06T00:27:08.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261626+0000", 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072235140000000003 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261643+0000", 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6441000000000001e-05 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2557 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.492113+0000", 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1768174180000006, 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058679479999999996, 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2557 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.746 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.492113+0000", 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.492113+0000", 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.492115+0000", 2024-05-06T00:27:08.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4930000000000001e-06 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.492128+0000", 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2364000000000001e-05 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.492158+0000", 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0035999999999998e-05 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.492173+0000", 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5109999999999999e-05 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.492294+0000", 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121693 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.492438+0000", 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014351500000000001 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.493120+0000", 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068213099999999999 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.494963+0000", 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018426130000000001 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.497817+0000", 2024-05-06T00:27:08.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028546539999999999 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.497908+0000", 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1119000000000006e-05 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.497981+0000", 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2219999999999996e-05 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2574 3.4 3:218edb7b:::rbd_data.110cc8beeb04.0000000000000007:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1660933470000003, 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006796074, 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2574 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.752 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7329999999999999e-06 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502852+0000", 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2203e-05 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502884+0000", 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2543000000000003e-05 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502906+0000", 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2027000000000001e-05 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503066+0000", 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159946 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503260+0000", 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019375100000000001 2024-05-06T00:27:08.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503688+0000", 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042773399999999998 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.506368+0000", 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002680689 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509377+0000", 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003008667 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509527+0000", 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149498 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509633+0000", 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.757 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1279 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.007816107, 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084436479999999998, 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:08.758 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1279 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661117+0000", 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6929999999999999e-06 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661132+0000", 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5153999999999999e-05 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661170+0000", 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7886000000000003e-05 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661188+0000", 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7816e-05 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661295+0000", 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001072 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661597+0000", 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030253000000000002 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.662320+0000", 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072273200000000002 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669339+0000", 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070190219999999998 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669377+0000", 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7935999999999997e-05 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669495+0000", 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117599 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669558+0000", 2024-05-06T00:27:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3079999999999999e-05 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1293 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.0055085479999999, 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066635549999999998, 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.764 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1293 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663424+0000", 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6599999999999999e-06 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663439+0000", 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4633e-05 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663471+0000", 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1986999999999997e-05 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663491+0000", 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0435999999999999e-05 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663643+0000", 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151929 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663863+0000", 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021996200000000001 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.664839+0000", 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00097599500000000001 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669583+0000", 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047435070000000001 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669694+0000", 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011181899999999999 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669963+0000", 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026898800000000001 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670085+0000", 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012163900000000001 2024-05-06T00:27:08.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1294 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.0052086239999998, 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086615830000000005, 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1294 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:08.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663724+0000", 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.61e-06 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663739+0000", 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.447e-05 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663768+0000", 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9957000000000001e-05 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663789+0000", 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0420000000000001e-05 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.667197+0000", 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003407963 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.667372+0000", 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000175532 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.668276+0000", 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00090397299999999997 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670141+0000", 2024-05-06T00:27:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001864239 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672093+0000", 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019525650000000001 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672271+0000", 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017802900000000001 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672383+0000", 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011182499999999999 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1302 3.4 3:214da2ae:::rbd_data.110c7776112a.0000000000000007:head [set-alloc-hint object_size 33554432 write_size 33554432,write 1286144~60416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.669804+0000", 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9991260070000001, 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060376470000000002, 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1302 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669804+0000", 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669804+0000", 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669807+0000", 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7760000000000002e-06 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669899+0000", 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1812999999999997e-05 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669924+0000", 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4973e-05 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669943+0000", 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9582999999999999e-05 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670094+0000", 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151149 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670227+0000", 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000132965 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670436+0000", 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020880900000000001 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.675634+0000", 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051973610000000002 2024-05-06T00:27:08.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.675650+0000", 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6707000000000001e-05 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.675760+0000", 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010966100000000001 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.675842+0000", 2024-05-06T00:27:08.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1849999999999997e-05 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1303 3.4 3:214da2ae:::rbd_data.110c7776112a.0000000000000007:head [set-alloc-hint object_size 33554432 write_size 33554432,write 2125824~23552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9989383929999995, 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063999010000000004, 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1303 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:08.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669994+0000", 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8169999999999999e-06 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670033+0000", 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9113000000000003e-05 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670057+0000", 2024-05-06T00:27:08.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4334000000000001e-05 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670078+0000", 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1539000000000002e-05 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672106+0000", 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00202797 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672301+0000", 2024-05-06T00:27:08.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019495800000000001 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672680+0000", 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037862600000000002 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.675864+0000", 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003183745 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:08.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676177+0000", 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031343499999999998 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676313+0000", 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013609700000000001 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676392+0000", 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8267000000000001e-05 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:08.788 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:08.788 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:08.789 INFO:teuthology.orchestra.run.smithi186.stdout:937 read 0x2593000 thru 0x25930b9 (0xba bytes) 2024-05-06T00:27:08.789 INFO:teuthology.orchestra.run.smithi186.stdout:938 write 0xc409000 thru 0xc412ad8 (0x9ad9 bytes) 2024-05-06T00:27:08.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:08.790 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:08.785+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:08.834 INFO:teuthology.orchestra.run.smithi186.stdout:940 trunc from 0xc412c00 to 0x533400 2024-05-06T00:27:08.889 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:08.889+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:08.914 INFO:teuthology.orchestra.run.smithi186.stdout:941 read 0x38e200 thru 0x39c24a (0xe04b bytes) 2024-05-06T00:27:08.944 INFO:teuthology.orchestra.run.smithi186.stdout:942 trunc from 0x533400 to 0xb7eda00 2024-05-06T00:27:08.967 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:08.968 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:08.968 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:08.968 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:08.968 INFO:teuthology.orchestra.run.smithi186.stdout:943 write 0xb384400 thru 0xb38ff4c (0xbb4d bytes) 2024-05-06T00:27:08.969 INFO:teuthology.orchestra.run.smithi186.stdout:945 punch from 0x1383800 to 0x138d600, (0x9e00 bytes) 2024-05-06T00:27:08.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:08.995 INFO:teuthology.orchestra.run.smithi186.stdout:946 trunc from 0xb7eda00 to 0x120de00 2024-05-06T00:27:09.026 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:09.025+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:09.077 INFO:teuthology.orchestra.run.smithi186.stdout:949 write 0xa13c800 thru 0xa14020d (0x3a0e bytes) 2024-05-06T00:27:09.078 INFO:teuthology.orchestra.run.smithi186.stdout:950 write 0x4112e00 thru 0x411f49c (0xc69d bytes) 2024-05-06T00:27:09.079 INFO:teuthology.orchestra.run.smithi186.stdout:951 read 0x5fa8e00 thru 0x5fb8b2d (0xfd2e bytes) 2024-05-06T00:27:09.079 INFO:teuthology.orchestra.run.smithi186.stdout:952 punch from 0x3a1b000 to 0x3a28a00, (0xda00 bytes) 2024-05-06T00:27:09.080 INFO:teuthology.orchestra.run.smithi186.stdout:953 read 0x38ade00 thru 0x38af6c6 (0x18c7 bytes) 2024-05-06T00:27:09.080 INFO:teuthology.orchestra.run.smithi186.stdout:954 punch from 0xccc800 to 0xcd6000, (0x9800 bytes) 2024-05-06T00:27:09.081 INFO:teuthology.orchestra.run.smithi186.stdout:956 write 0x98ebc00 thru 0x98ed83e (0x1c3f bytes) 2024-05-06T00:27:09.081 INFO:teuthology.orchestra.run.smithi186.stdout:958 read 0x43cfe00 thru 0x43d5d10 (0x5f11 bytes) 2024-05-06T00:27:09.081 INFO:teuthology.orchestra.run.smithi186.stdout:960 write 0x3408c00 thru 0x3414fbe (0xc3bf bytes) 2024-05-06T00:27:09.081 INFO:teuthology.orchestra.run.smithi186.stdout:964 read 0x5884600 thru 0x5893c11 (0xf612 bytes) 2024-05-06T00:27:09.081 INFO:teuthology.orchestra.run.smithi186.stdout:967 read 0x4335800 thru 0x433afca (0x57cb bytes) 2024-05-06T00:27:09.082 INFO:teuthology.orchestra.run.smithi186.stdout:969 punch from 0xa013800 to 0xa013a00, (0x200 bytes) 2024-05-06T00:27:09.091 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:09.089+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:09.094 INFO:teuthology.orchestra.run.smithi186.stdout:971 trunc from 0xa140400 to 0x4317600 2024-05-06T00:27:09.147 INFO:teuthology.orchestra.run.smithi186.stdout:972 trunc from 0x4317600 to 0x888f000 2024-05-06T00:27:09.157 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:09.157 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:09.157 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:09.158 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:09.158 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:09.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:09.192 INFO:teuthology.orchestra.run.smithi186.stdout:973 trunc from 0x888f000 to 0x320e00 2024-05-06T00:27:09.213 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:09.213+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:09.231 INFO:teuthology.orchestra.run.smithi186.stdout:974 read 0x5b400 thru 0x5e781 (0x3382 bytes) 2024-05-06T00:27:09.236 INFO:teuthology.orchestra.run.smithi186.stdout:976 punch from 0x56200 to 0x61000, (0xae00 bytes) 2024-05-06T00:27:09.236 INFO:teuthology.orchestra.run.smithi186.stdout:977 clone 8 order 21 su 1048576 sc 3 2024-05-06T00:27:09.293 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:09.292+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:09.341 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:223 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3702784~58880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.614257653999999, 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005189656, 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 223 2024-05-06T00:27:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723867+0000", 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8569999999999999e-06 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724065+0000", 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019875599999999999 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724102+0000", 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6362999999999997e-05 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724122+0000", 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0519000000000001e-05 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724220+0000", 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7466000000000005e-05 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724367+0000", 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014768100000000001 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724516+0000", 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014873800000000001 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728904+0000", 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043882369999999997 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728924+0000", 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9959999999999999e-05 2024-05-06T00:27:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728992+0000", 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7519000000000002e-05 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729053+0000", 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.156e-05 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:224 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2300416~15872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.613942439000001, 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062082700000000001, 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 224 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724180+0000", 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.471e-06 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724206+0000", 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729e-05 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724221+0000", 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5114e-05 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724230+0000", 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2369999999999998e-06 2024-05-06T00:27:09.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725101+0000", 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087058999999999997 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725503+0000", 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000402474 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725876+0000", 2024-05-06T00:27:09.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037291499999999998 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729081+0000", 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032042339999999998 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730250+0000", 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001168955 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730326+0000", 2024-05-06T00:27:09.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6148000000000003e-05 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730387+0000", 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1402999999999995e-05 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:225 3.7 3:e39c62f0:::rbd_data.110cc8beeb04.0000000000000012:head [set-alloc-hint object_size 4194304 write_size 4194304,write 905216~12288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.613872110999999, 2024-05-06T00:27:09.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062774959999999996, 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 225 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724250+0000", 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.71e-07 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724265+0000", 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.502e-05 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724275+0000", 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7569999999999999e-06 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724282+0000", 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1860000000000003e-06 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.727270+0000", 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029884489999999998 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.727639+0000", 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036815399999999999 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728041+0000", 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000402013 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729289+0000", 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012479399999999999 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730444+0000", 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001155498 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730493+0000", 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8727999999999998e-05 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730527+0000", 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3880000000000001e-05 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:298 3.4 e28/18 3:26d09f8b:::rbd_object_map.110cc8beeb04:head v 28'28, mlcod=28'24)", 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.574825578, 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048531540000000001, 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763298+0000", 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9630000000000001e-06 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763310+0000", 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2782999999999999e-05 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763333+0000", 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2025e-05 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763351+0000", 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8882999999999999e-05 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763872+0000", 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00052017299999999997 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763924+0000", 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2473999999999999e-05 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.767937+0000", 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040133399999999998 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768149+0000", 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021151300000000001 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:322 3.7 3:e39c62f0:::rbd_data.110cc8beeb04.0000000000000012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.547151408, 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050787829999999999, 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 322 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:27:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790972+0000", 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.731e-06 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790977+0000", 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.933e-06 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791004+0000", 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6974999999999998e-05 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791012+0000", 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6070000000000008e-06 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791099+0000", 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6571999999999997e-05 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791197+0000", 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8072000000000006e-05 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791302+0000", 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105414 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.792824+0000", 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001521912 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795944+0000", 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031202629999999999 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796002+0000", 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7547999999999999e-05 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796049+0000", 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6755999999999998e-05 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:348 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.541790106000001, 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048444150000000004, 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.369 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 348 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796332+0000", 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0240000000000001e-06 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796336+0000", 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2710000000000003e-06 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796348+0000", 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1843e-05 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796355+0000", 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9129999999999997e-06 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796409+0000", 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3698e-05 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2955999999999997e-05 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796581+0000", 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9319000000000002e-05 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.798666+0000", 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020843089999999999 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801063+0000", 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002397518 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801141+0000", 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7948000000000007e-05 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801176+0000", 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4616000000000002e-05 2024-05-06T00:27:09.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:351 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.541639529999999, 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049431070000000004, 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 351 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.37e-07 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796485+0000", 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.582e-06 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796495+0000", 2024-05-06T00:27:09.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6350000000000001e-06 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796499+0000", 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3100000000000002e-06 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796827+0000", 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032834500000000001 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796904+0000", 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6808000000000001e-05 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797004+0000", 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9998e-05 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801185+0000", 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041808089999999997 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801284+0000", 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9228999999999995e-05 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801363+0000", 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9214000000000004e-05 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801425+0000", 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1639999999999999e-05 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:833 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [truncate 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.203859700000001, 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047565890000000003, 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 833 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:27:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2099999999999996e-07 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134265+0000", 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5909999999999998e-06 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134273+0000", 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5599999999999994e-06 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134279+0000", 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2580000000000003e-06 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134801+0000", 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00052267500000000001 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134865+0000", 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3507e-05 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134939+0000", 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4564e-05 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.136596+0000", 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016564749999999999 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.138912+0000", 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002316214 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.138974+0000", 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1941999999999997e-05 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.139018+0000", 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4181999999999998e-05 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1015 3.8 3:1391aa96:::rbd_data.110cc8beeb04.0000000000000025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:27:09.386 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.08464019, 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084885580000000002, 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253483+0000", 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0090000000000002e-06 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253489+0000", 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2639999999999997e-06 2024-05-06T00:27:09.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253510+0000", 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0462000000000002e-05 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253519+0000", 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5550000000000008e-06 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253594+0000", 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4345999999999996e-05 2024-05-06T00:27:09.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253675+0000", 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0860999999999997e-05 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253779+0000", 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104353 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.256496+0000", 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002716703 2024-05-06T00:27:09.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261853+0000", 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053571549999999997 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261907+0000", 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4381e-05 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261970+0000", 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2469000000000001e-05 2024-05-06T00:27:09.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1017 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.084576992000001, 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084085759999999992, 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1017 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253545+0000", 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6000000000000003e-07 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253549+0000", 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6710000000000001e-06 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253557+0000", 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253563+0000", 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9440000000000002e-06 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253683+0000", 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120147 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253759+0000", 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5817999999999998e-05 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253858+0000", 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9053000000000002e-05 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259253+0000", 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005395047 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261834+0000", 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025814929999999998 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261900+0000", 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5738e-05 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261953+0000", 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2784000000000001e-05 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1018 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.397 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.084539783, 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085355099999999996, 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4199999999999996e-07 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253587+0000", 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7119999999999999e-06 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253595+0000", 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7980000000000006e-06 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253601+0000", 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254070+0000", 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00046893599999999998 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254144+0000", 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4943000000000004e-05 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254244+0000", 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9091000000000004e-05 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259286+0000", 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050428850000000004 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262019+0000", 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027324430000000002 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262072+0000", 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2994000000000001e-05 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262117+0000", 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5155999999999999e-05 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1022 3.8 3:17fe3906:::rbd_data.110cc8beeb04.0000000000000036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.084512176, 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085045799999999994, 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1022 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253610+0000", 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.51e-07 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253612+0000", 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2979999999999999e-06 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253618+0000", 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.426e-06 2024-05-06T00:27:09.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253623+0000", 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7260000000000002e-06 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254047+0000", 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042410499999999998 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254100+0000", 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2592000000000001e-05 2024-05-06T00:27:09.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254190+0000", 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0001999999999997e-05 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259272+0000", 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050825840000000002 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262019+0000", 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00274654 2024-05-06T00:27:09.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262071+0000", 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2091e-05 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262114+0000", 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2664999999999999e-05 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1042 3.1e 3:7e323eed:::rbd_data.110cc8beeb04.0000000000000002:head [stat,truncate 3670016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.073017781000001, 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059170580000000002, 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1042 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265105+0000", 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.857e-06 2024-05-06T00:27:09.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265112+0000", 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7769999999999997e-06 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265131+0000", 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8805e-05 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265144+0000", 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3474e-05 2024-05-06T00:27:09.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265260+0000", 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000115785 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265343+0000", 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3010000000000007e-05 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265472+0000", 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128625 2024-05-06T00:27:09.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.266670+0000", 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001198382 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.270924+0000", 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042537260000000002 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.270974+0000", 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0484e-05 2024-05-06T00:27:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271021+0000", 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6133e-05 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1045 3.1e 3:7ff04bbe:::rbd_data.110cc8beeb04.0000000000000005:head [stat,truncate 3538944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.072948555, 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005939003, 2024-05-06T00:27:09.414 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1045 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2500000000000005e-07 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265177+0000", 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3440000000000001e-06 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265187+0000", 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755e-05 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265194+0000", 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3999999999999997e-06 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265713+0000", 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051880999999999997 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265781+0000", 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8140999999999997e-05 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265878+0000", 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7052999999999994e-05 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.266675+0000", 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00079742599999999995 2024-05-06T00:27:09.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271049+0000", 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043737630000000001 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271077+0000", 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.775e-05 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271112+0000", 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4935999999999999e-05 2024-05-06T00:27:09.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1205 3.4 e29/18 3:26d09f8b:::rbd_object_map.110cc8beeb04:head v 29'87, mlcod=29'83)", 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.958284804, 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043375489999999996, 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379843+0000", 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9789999999999998e-06 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379858+0000", 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4946999999999999e-05 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379865+0000", 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.754e-06 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379942+0000", 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6687000000000006e-05 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379960+0000", 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8357999999999999e-05 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.384139+0000", 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041789239999999997 2024-05-06T00:27:09.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.384174+0000", 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5221e-05 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2561 3.1e 3:7e323eed:::rbd_data.110cc8beeb04.0000000000000002:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498449+0000", 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.8396727160000008, 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068185010000000002, 2024-05-06T00:27:09.424 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2561 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498449+0000", 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498449+0000", 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498451+0000", 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3860000000000001e-06 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498464+0000", 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3477e-05 2024-05-06T00:27:09.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498506+0000", 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1596e-05 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498565+0000", 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9212999999999997e-05 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498740+0000", 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000174902 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.499058+0000", 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031793400000000002 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.499554+0000", 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00049580299999999996 2024-05-06T00:27:09.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503646+0000", 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040925689999999999 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503684+0000", 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7209999999999998e-05 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503815+0000", 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131142 2024-05-06T00:27:09.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505267+0000", 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001452269 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2567 3.1e 3:7ff04bbe:::rbd_data.110cc8beeb04.0000000000000005:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.8394659680000007, 2024-05-06T00:27:09.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091482830000000001, 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2567 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498657+0000", 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.75e-06 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498666+0000", 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3030000000000005e-06 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498690+0000", 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-05 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498707+0000", 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7326999999999999e-05 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501155+0000", 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002447884 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501468+0000", 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031281100000000002 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501983+0000", 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051484999999999996 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505388+0000", 2024-05-06T00:27:09.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034055240000000001 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507550+0000", 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021622690000000001 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507694+0000", 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014377899999999999 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507804+0000", 2024-05-06T00:27:09.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109486 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2582 3.7 3:e05e3ccf:::rbd_data.110cc8beeb04.000000000000003b:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.835282393, 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062205380000000003, 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2582 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502841+0000", 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.497e-06 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502852+0000", 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0246e-05 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502884+0000", 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2722999999999998e-05 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502924+0000", 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9657000000000003e-05 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503194+0000", 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026947100000000002 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503406+0000", 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021234499999999999 2024-05-06T00:27:09.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503867+0000", 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00046143300000000003 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508399+0000", 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045314639999999998 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508816+0000", 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041726699999999997 2024-05-06T00:27:09.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508956+0000", 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000139849 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509059+0000", 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010358599999999999 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2583 3.8 3:1592c298:::rbd_data.110cc8beeb04.000000000000003c:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.503003+0000", 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.8351186310000003, 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067545779999999998, 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2583 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.442 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503003+0000", 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503003+0000", 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503005+0000", 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9400000000000001e-06 2024-05-06T00:27:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503029+0000", 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4695999999999999e-05 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503057+0000", 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8017e-05 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503076+0000", 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8516999999999999e-05 2024-05-06T00:27:09.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503228+0000", 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015184500000000001 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503365+0000", 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013740900000000001 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503789+0000", 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042415600000000002 2024-05-06T00:27:09.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508615+0000", 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048255570000000003 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509494+0000", 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087899899999999999 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509642+0000", 2024-05-06T00:27:09.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000148283 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509757+0000", 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000115159 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:2578 3.9 e32/18 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head v 32'23, mlcod=29'20)", 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.503523+0000", 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.8345988020000004, 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073103200000000004, 2024-05-06T00:27:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503523+0000", 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503523+0000", 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503523+0000", 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.23e-07 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503535+0000", 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1297e-05 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503557+0000", 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2532999999999999e-05 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503577+0000", 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0264000000000001e-05 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503844+0000", 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000266171 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503892+0000", 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8192999999999998e-05 2024-05-06T00:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510706+0000", 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068146730000000003 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510833+0000", 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012636599999999999 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:09.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:09.454 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:09.452+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:09.554 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:09.553+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:09.584 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:09.584 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:09.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:09.655 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:09.653+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:09.715 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:09.715 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:09.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:09.756 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:09.753+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:09.854 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:09.854 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:09.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:09.854 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:09.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.854 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.854 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.250258066000001, 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:09.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.471428281000001, 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012319788, 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381231+0000", 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.624e-06 2024-05-06T00:27:09.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381243+0000", 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2476e-05 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381263+0000", 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-05 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381278+0000", 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5183e-05 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389490+0000", 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082113770000000006 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389614+0000", 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012416900000000001 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389813+0000", 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199321 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391953+0000", 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021402869999999998 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393301+0000", 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001347729 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393438+0000", 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136666 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393549+0000", 2024-05-06T00:27:09.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110856 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.471352478, 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014641942999999999, 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381306+0000", 2024-05-06T00:27:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.403e-06 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381316+0000", 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381341+0000", 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4372999999999999e-05 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381356+0000", 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5207e-05 2024-05-06T00:27:09.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390708+0000", 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093519799999999993 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390856+0000", 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839200000000001 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391064+0000", 2024-05-06T00:27:09.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000207302 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394636+0000", 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003572563 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395813+0000", 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011765910000000001 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395885+0000", 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2601999999999994e-05 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395947+0000", 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1183e-05 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.471275802000001, 2024-05-06T00:27:09.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017147241000000001, 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 47 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:09.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381383+0000", 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6199999999999999e-06 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381504+0000", 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120709 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381525+0000", 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1705999999999999e-05 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381541+0000", 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5367e-05 2024-05-06T00:27:09.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392051+0000", 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010510442 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392177+0000", 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012575600000000001 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392387+0000", 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020954799999999999 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394744+0000", 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023578319999999998 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398377+0000", 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036325189999999999 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398462+0000", 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4668999999999999e-05 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398529+0000", 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7073e-05 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:09.877 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.471075786999997, 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017264267, 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 48 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:09.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381583+0000", 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7069999999999999e-06 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381594+0000", 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1365999999999999e-05 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381618+0000", 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4014000000000001e-05 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381639+0000", 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0616e-05 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393664+0000", 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012025179 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393804+0000", 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014017599999999999 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393998+0000", 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019390499999999999 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398573+0000", 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045742789999999997 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398683+0000", 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110136 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398786+0000", 2024-05-06T00:27:09.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103259 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398846+0000", 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9630000000000003e-05 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.470984524000002, 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018281285000000001, 2024-05-06T00:27:09.883 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 49 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381674+0000", 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.559e-06 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381685+0000", 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1049999999999999e-05 2024-05-06T00:27:09.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381705+0000", 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0227000000000001e-05 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381728+0000", 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2467e-05 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394829+0000", 2024-05-06T00:27:09.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013101358 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394953+0000", 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012340499999999999 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395145+0000", 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019252900000000001 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399714+0000", 2024-05-06T00:27:09.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045683620000000003 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399788+0000", 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4719000000000001e-05 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399871+0000", 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3092999999999999e-05 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399954+0000", 2024-05-06T00:27:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2515999999999994e-05 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.470894844999997, 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020991408, 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 50 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381764+0000", 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381894+0000", 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012972600000000001 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381927+0000", 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3399000000000003e-05 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381938+0000", 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0677e-05 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396032+0000", 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014094257000000001 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396199+0000", 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016722899999999999 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396415+0000", 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021597200000000001 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.400244+0000", 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038293009999999998 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402502+0000", 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002257972 2024-05-06T00:27:09.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402645+0000", 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014225199999999999 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402754+0000", 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010917300000000001 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.470697469999998, 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021178922999999999, 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 51 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.895 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381961+0000", 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:27:09.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381968+0000", 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.007e-06 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381979+0000", 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1197e-05 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381987+0000", 2024-05-06T00:27:09.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9629999999999998e-06 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397517+0000", 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015529589999999999 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397635+0000", 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011846899999999999 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397830+0000", 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019498200000000001 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402867+0000", 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050367240000000002 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403002+0000", 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134609 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403082+0000", 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0666000000000003e-05 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403139+0000", 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6462999999999998e-05 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:09.900 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.470656712999997, 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021264655, 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:09.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382001+0000", 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7300000000000005e-07 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382007+0000", 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000004e-06 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382017+0000", 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9799999999999993e-06 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382029+0000", 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1420000000000001e-05 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398910+0000", 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016881075999999998 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399069+0000", 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015880199999999999 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399327+0000", 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025800200000000001 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402970+0000", 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003643725 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403155+0000", 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018459100000000001 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403221+0000", 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6106999999999996e-05 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403265+0000", 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4135999999999999e-05 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.267059394999997, 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012456859000000001, 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.906 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:09.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585599+0000", 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5899999999999995e-07 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585692+0000", 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3398000000000003e-05 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585704+0000", 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2318e-05 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585713+0000", 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7649999999999999e-06 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593668+0000", 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079544980000000008 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593723+0000", 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5914e-05 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593815+0000", 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1583999999999996e-05 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595554+0000", 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017386190000000001 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.597968+0000", 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024139500000000002 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598008+0000", 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9962000000000001e-05 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598055+0000", 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7092000000000002e-05 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.266927137000003, 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013009979, 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:09.912 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 76 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585731+0000", 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0999999999999997e-07 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585738+0000", 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6059999999999999e-06 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585752+0000", 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4248e-05 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585762+0000", 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8660000000000001e-06 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594480+0000", 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087185100000000005 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594616+0000", 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135375 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594806+0000", 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019038099999999999 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595986+0000", 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011804039999999999 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598676+0000", 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026891889999999998 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598710+0000", 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4471999999999998e-05 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598740+0000", 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0018e-05 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:401 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 786432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.917 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.043440895, 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012526862999999999, 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 401 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809218+0000", 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:09.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809224+0000", 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.037e-06 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809242+0000", 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809256+0000", 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4027e-05 2024-05-06T00:27:09.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821381+0000", 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012125268999999999 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821510+0000", 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129219 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821743+0000", 2024-05-06T00:27:09.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023277500000000001 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:402 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 1703936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.043378942, 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013133265, 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 402 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809279+0000", 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809285+0000", 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809303+0000", 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7983e-05 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809317+0000", 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3856000000000001e-05 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821841+0000", 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012524400999999999 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822066+0000", 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022526100000000001 2024-05-06T00:27:09.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822412+0000", 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034507400000000001 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:403 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 2621440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.04332286, 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013546829999999999, 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 403 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809335+0000", 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1200000000000001e-06 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809341+0000", 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9200000000000001e-06 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809360+0000", 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8184000000000001e-05 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809373+0000", 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822550+0000", 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013176998000000001 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822703+0000", 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015292600000000001 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822881+0000", 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017814600000000001 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:404 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 3538944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.930 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.043223429999999, 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013816116999999999, 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 404 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:09.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809435+0000", 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4300000000000001e-06 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809441+0000", 2024-05-06T00:27:09.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9730000000000002e-06 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809461+0000", 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9786e-05 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809498+0000", 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6706999999999999e-05 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822957+0000", 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013459296000000001 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823087+0000", 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054200000000001 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823250+0000", 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016238299999999999 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:405 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 4456448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.043137766999999, 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014023473999999999, 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 405 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809521+0000", 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3629999999999999e-06 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809527+0000", 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1369999999999999e-06 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809544+0000", 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7516e-05 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809559+0000", 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.455e-05 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823295+0000", 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013736474 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823397+0000", 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010108899999999999 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823543+0000", 2024-05-06T00:27:09.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014634500000000001 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:406 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 5373952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.043079345000001, 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01426091, 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 406 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809579+0000", 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.077e-06 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809584+0000", 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3530000000000002e-06 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809604+0000", 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9962999999999998e-05 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809632+0000", 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7710000000000001e-05 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823589+0000", 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013957476 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823695+0000", 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010535900000000001 2024-05-06T00:27:09.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823839+0000", 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143972 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:407 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 6291456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.043005308, 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014464551000000001, 2024-05-06T00:27:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 407 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809653+0000", 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2330000000000001e-06 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809659+0000", 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0170000000000002e-06 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809678+0000", 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8703e-05 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809692+0000", 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4326e-05 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823880+0000", 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014187936999999999 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823982+0000", 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102266 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824116+0000", 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013406900000000001 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:408 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.042929279000001, 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014667517, 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 408 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809729+0000", 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5430000000000001e-06 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809749+0000", 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9227000000000001e-05 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809769+0000", 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0556000000000001e-05 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809784+0000", 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4507e-05 2024-05-06T00:27:09.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824158+0000", 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014374418999999999 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824258+0000", 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100209 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824395+0000", 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013705600000000001 2024-05-06T00:27:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:409 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.042853653, 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014988434, 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:09.952 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 409 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809805+0000", 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.917e-06 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809812+0000", 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5930000000000002e-06 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809830+0000", 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8159e-05 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809844+0000", 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3764000000000001e-05 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824478+0000", 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014633703 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824611+0000", 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132893 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:09.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824792+0000", 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018140499999999999 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:09.956 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:09.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:27:09.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:09.958 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:09.956+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:09.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:10.018 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone7 from 0x320e00 (overlap 0x320e00) to 0x0 2024-05-06T00:27:10.059 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:10.057+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:10.131 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:10.131 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:10.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:10.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:10.148 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:10.160 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:10.157+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:10.165 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:10.165 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:10.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:10.182 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:10.199 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:10.199 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:10.219 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:10.219 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:10.239 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:10.261 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:10.260+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:10.274 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:10.274 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:10.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:10.380 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:10.380+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:10.430 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-05-06T00:27:10.455 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:10.456 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:10.466 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:10.460+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:10.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:10.564 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:10.561+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:10.652 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:10.652 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:10.665 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:10.664+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:10.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:10.766 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:10.764+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:10.837 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:10.837 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:10.852 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:10.868 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:10.865+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:10.872 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:27:10.968 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:10.965+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:11.029 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:11.030 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:11.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:11.069 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:11.068+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:11.170 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:11.168+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:11.190 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:11.190 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:11.206 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:11.271 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:11.269+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:11.336 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:11.336 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:11.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:11.397 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:11.397+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:11.473 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:11.469+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:11.522 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:11.522 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:11.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:11.577 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:11.577+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:11.674 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:11.672+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:11.699 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:11.699 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:11.706 INFO:teuthology.orchestra.run.smithi186.stdout:980 write 0xdcb6c00 thru 0xdcbc99f (0x5da0 bytes) 2024-05-06T00:27:11.708 INFO:teuthology.orchestra.run.smithi186.stdout:981 write 0x2e03e00 thru 0x2e09c90 (0x5e91 bytes) 2024-05-06T00:27:11.708 INFO:teuthology.orchestra.run.smithi186.stdout:983 punch from 0x8cce200 to 0x8cda000, (0xbe00 bytes) 2024-05-06T00:27:11.709 INFO:teuthology.orchestra.run.smithi186.stdout:984 write 0xcc8c200 thru 0xcc8e876 (0x2677 bytes) 2024-05-06T00:27:11.715 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:11.748 INFO:teuthology.orchestra.run.smithi186.stdout:985 write 0xdcf1e00 thru 0xdcf4fa1 (0x31a2 bytes) 2024-05-06T00:27:11.769 INFO:teuthology.orchestra.run.smithi186.stdout:988 trunc from 0xdcf5000 to 0x6374a00 2024-05-06T00:27:11.775 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:11.772+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:11.807 INFO:teuthology.orchestra.run.smithi186.stdout:991 read 0x636aa00 thru 0x636eff3 (0x45f4 bytes) 2024-05-06T00:27:11.808 INFO:teuthology.orchestra.run.smithi186.stdout:992 read 0x844800 thru 0x84d119 (0x891a bytes) 2024-05-06T00:27:11.808 INFO:teuthology.orchestra.run.smithi186.stdout:993 write 0x47b2200 thru 0x47b59c8 (0x37c9 bytes) 2024-05-06T00:27:11.809 INFO:teuthology.orchestra.run.smithi186.stdout:994 write 0x479e000 thru 0x47ad783 (0xf784 bytes) 2024-05-06T00:27:11.839 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:11.839 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:11.841 INFO:teuthology.orchestra.run.smithi186.stdout:997 write 0x7c2bc00 thru 0x7c3b047 (0xf448 bytes) 2024-05-06T00:27:11.845 INFO:teuthology.orchestra.run.smithi186.stdout:999 trunc from 0x7c3b200 to 0x7e53a00 2024-05-06T00:27:11.854 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:11.872 INFO:teuthology.orchestra.run.smithi186.stdout:1000 write 0x5f3cc00 thru 0x5f46a9a (0x9e9b bytes) 2024-05-06T00:27:11.876 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:11.876+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:11.913 INFO:teuthology.orchestra.run.smithi186.stdout:1003 write 0xbd41c00 thru 0xbd4f95a (0xdd5b bytes) 2024-05-06T00:27:11.916 INFO:teuthology.orchestra.run.smithi186.stdout:1005 write 0x4eb000 thru 0x4f1216 (0x6217 bytes) 2024-05-06T00:27:11.922 INFO:teuthology.orchestra.run.smithi186.stdout:1006 trunc from 0xbd4fa00 to 0x9b10e00 2024-05-06T00:27:11.961 INFO:teuthology.orchestra.run.smithi186.stdout:1007 punch from 0x3d51800 to 0x3d56800, (0x5000 bytes) 2024-05-06T00:27:11.963 INFO:teuthology.orchestra.run.smithi186.stdout:1010 write 0x8ff9a00 thru 0x900643a (0xca3b bytes) 2024-05-06T00:27:11.964 INFO:teuthology.orchestra.run.smithi186.stdout:1011 write 0x36b4200 thru 0x36c0cc2 (0xcac3 bytes) 2024-05-06T00:27:11.964 INFO:teuthology.orchestra.run.smithi186.stdout:1012 read 0x6eae400 thru 0x6eba54b (0xc14c bytes) 2024-05-06T00:27:11.977 INFO:teuthology.orchestra.run.smithi186.stdout:1013 trunc from 0x9b10e00 to 0x36aa800 2024-05-06T00:27:11.977 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:11.976+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:12.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:12.044 INFO:teuthology.orchestra.run.smithi186.stdout:1014 write 0x5ee0000 thru 0x5eefdd8 (0xfdd9 bytes) 2024-05-06T00:27:12.046 INFO:teuthology.orchestra.run.smithi186.stdout:1015 punch from 0x3dcce00 to 0x3dd0e00, (0x4000 bytes) 2024-05-06T00:27:12.074 INFO:teuthology.orchestra.run.smithi186.stdout:1016 write 0x5faaa00 thru 0x5fb3f7f (0x9580 bytes) 2024-05-06T00:27:12.075 INFO:teuthology.orchestra.run.smithi186.stdout:1017 punch from 0x5e0cc00 to 0x5e13000, (0x6400 bytes) 2024-05-06T00:27:12.078 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:12.076+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:12.093 INFO:teuthology.orchestra.run.smithi186.stdout:1018 trunc from 0x5fb4000 to 0xc22b600 2024-05-06T00:27:12.115 INFO:teuthology.orchestra.run.smithi186.stdout:1020 read 0x882ac00 thru 0x882ba84 (0xe85 bytes) 2024-05-06T00:27:12.117 INFO:teuthology.orchestra.run.smithi186.stdout:1021 read 0xb7d3a00 thru 0xb7dcce2 (0x92e3 bytes) 2024-05-06T00:27:12.117 INFO:teuthology.orchestra.run.smithi186.stdout:1022 read 0x6529800 thru 0x6534f39 (0xb73a bytes) 2024-05-06T00:27:12.117 INFO:teuthology.orchestra.run.smithi186.stdout:1023 write 0x5672200 thru 0x5675b61 (0x3962 bytes) 2024-05-06T00:27:12.117 INFO:teuthology.orchestra.run.smithi186.stdout:1024 read 0x232c800 thru 0x232f77f (0x2f80 bytes) 2024-05-06T00:27:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:12.145 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:12.148 INFO:teuthology.orchestra.run.smithi186.stdout:1027 write 0xc657800 thru 0xc663163 (0xb964 bytes) 2024-05-06T00:27:12.150 INFO:teuthology.orchestra.run.smithi186.stdout:1029 read 0x1944800 thru 0x195411f (0xf920 bytes) 2024-05-06T00:27:12.150 INFO:teuthology.orchestra.run.smithi186.stdout:1031 read 0x9563e00 thru 0x956f0fb (0xb2fc bytes) 2024-05-06T00:27:12.150 INFO:teuthology.orchestra.run.smithi186.stdout:1032 write 0xa72ce00 thru 0xa736594 (0x9795 bytes) 2024-05-06T00:27:12.151 INFO:teuthology.orchestra.run.smithi186.stdout:1037 write 0x4054000 thru 0x4058a3e (0x4a3f bytes) 2024-05-06T00:27:12.152 INFO:teuthology.orchestra.run.smithi186.stdout:1038 write 0x146c000 thru 0x147aafd (0xeafe bytes) 2024-05-06T00:27:12.152 INFO:teuthology.orchestra.run.smithi186.stdout:1040 write 0x9844000 thru 0x9853195 (0xf196 bytes) 2024-05-06T00:27:12.152 INFO:teuthology.orchestra.run.smithi186.stdout:1041 read 0xe74000 thru 0xe7f1b8 (0xb1b9 bytes) 2024-05-06T00:27:12.153 INFO:teuthology.orchestra.run.smithi186.stdout:1043 read 0x2547a00 thru 0x2556856 (0xee57 bytes) 2024-05-06T00:27:12.153 INFO:teuthology.orchestra.run.smithi186.stdout:1044 read 0x1eec400 thru 0x1eeccd2 (0x8d3 bytes) 2024-05-06T00:27:12.153 INFO:teuthology.orchestra.run.smithi186.stdout:1045 write 0x253b800 thru 0x253f2a6 (0x3aa7 bytes) 2024-05-06T00:27:12.153 INFO:teuthology.orchestra.run.smithi186.stdout:1046 read 0x1a26800 thru 0x1a2fbe2 (0x93e3 bytes) 2024-05-06T00:27:12.171 INFO:teuthology.orchestra.run.smithi186.stdout:1047 trunc from 0xc663200 to 0x3c06c00 2024-05-06T00:27:12.179 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:12.176+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:12.224 INFO:teuthology.orchestra.run.smithi186.stdout:1050 punch from 0x1422000 to 0x1422c00, (0xc00 bytes) 2024-05-06T00:27:12.226 INFO:teuthology.orchestra.run.smithi186.stdout:1053 read 0x369ec00 thru 0x36a3430 (0x4831 bytes) 2024-05-06T00:27:12.241 INFO:teuthology.orchestra.run.smithi186.stdout:1054 trunc from 0x3c06c00 to 0x8d89800 2024-05-06T00:27:12.261 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:12.280 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:12.280+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:12.281 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:12.281 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:12.296 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:12.296 INFO:teuthology.orchestra.run.smithi186.stdout:1055 write 0xc72b600 thru 0xc72c399 (0xd9a bytes) 2024-05-06T00:27:12.297 INFO:teuthology.orchestra.run.smithi186.stdout:1056 punch from 0x905600 to 0x907c00, (0x2600 bytes) 2024-05-06T00:27:12.306 INFO:teuthology.orchestra.run.smithi186.stdout:1057 trunc from 0xc72c400 to 0x99e3600 2024-05-06T00:27:12.335 INFO:teuthology.orchestra.run.smithi186.stdout:1058 write 0x3a05200 thru 0x3a149f3 (0xf7f4 bytes) 2024-05-06T00:27:12.348 INFO:teuthology.orchestra.run.smithi186.stdout:1059 trunc from 0x99e3600 to 0x4ed1400 2024-05-06T00:27:12.377 INFO:teuthology.orchestra.run.smithi186.stdout:1060 write 0x4a43200 thru 0x4a45737 (0x2538 bytes) 2024-05-06T00:27:12.381 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:12.380+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:12.417 INFO:teuthology.orchestra.run.smithi186.stdout:1061 write 0xa3b1200 thru 0xa3c0e5e (0xfc5f bytes) 2024-05-06T00:27:12.422 INFO:teuthology.orchestra.run.smithi186.stdout:1062 trunc from 0xa3c1000 to 0xbdcf400 2024-05-06T00:27:12.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:12.428 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:12.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:12.446 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:12.458 INFO:teuthology.orchestra.run.smithi186.stdout:1063 trunc from 0xbdcf400 to 0x70bba00 2024-05-06T00:27:12.477 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:12.478 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:12.481 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:223 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3702784~58880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.756790972000001, 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005189656, 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 223 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723864+0000", 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.723867+0000", 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8569999999999999e-06 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724065+0000", 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019875599999999999 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724102+0000", 2024-05-06T00:27:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6362999999999997e-05 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724122+0000", 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0519000000000001e-05 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724220+0000", 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7466000000000005e-05 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724367+0000", 2024-05-06T00:27:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014768100000000001 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724516+0000", 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014873800000000001 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728904+0000", 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043882369999999997 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728924+0000", 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9959999999999999e-05 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728992+0000", 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7519000000000002e-05 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729053+0000", 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.156e-05 2024-05-06T00:27:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:224 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2300416~15872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.756475757, 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062082700000000001, 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 224 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724179+0000", 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724180+0000", 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.471e-06 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724206+0000", 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729e-05 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724221+0000", 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5114e-05 2024-05-06T00:27:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724230+0000", 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2369999999999998e-06 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725101+0000", 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087058999999999997 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725503+0000", 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000402474 2024-05-06T00:27:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.725876+0000", 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037291499999999998 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729081+0000", 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032042339999999998 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730250+0000", 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001168955 2024-05-06T00:27:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730326+0000", 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6148000000000003e-05 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730387+0000", 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1402999999999995e-05 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:225 3.7 3:e39c62f0:::rbd_data.110cc8beeb04.0000000000000012:head [set-alloc-hint object_size 4194304 write_size 4194304,write 905216~12288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.756405429000001, 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062774959999999996, 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 225 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724249+0000", 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724250+0000", 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.71e-07 2024-05-06T00:27:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724265+0000", 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.502e-05 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724275+0000", 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7569999999999999e-06 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.724282+0000", 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1860000000000003e-06 2024-05-06T00:27:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.727270+0000", 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029884489999999998 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.727639+0000", 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036815399999999999 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.728041+0000", 2024-05-06T00:27:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000402013 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.729289+0000", 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012479399999999999 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730444+0000", 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001155498 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730493+0000", 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8727999999999998e-05 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.730527+0000", 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3880000000000001e-05 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:298 3.4 e28/18 3:26d09f8b:::rbd_object_map.110cc8beeb04:head v 28'28, mlcod=28'24)", 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:27:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.717358896, 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048531540000000001, 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763296+0000", 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763298+0000", 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9630000000000001e-06 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763310+0000", 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2782999999999999e-05 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763333+0000", 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2025e-05 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763351+0000", 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8882999999999999e-05 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763872+0000", 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00052017299999999997 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763924+0000", 2024-05-06T00:27:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2473999999999999e-05 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.767937+0000", 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040133399999999998 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768149+0000", 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021151300000000001 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:322 3.7 3:e39c62f0:::rbd_data.110cc8beeb04.0000000000000012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.689684725999999, 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050787829999999999, 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 322 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790970+0000", 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790972+0000", 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.731e-06 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.790977+0000", 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.933e-06 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791004+0000", 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6974999999999998e-05 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791012+0000", 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6070000000000008e-06 2024-05-06T00:27:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791099+0000", 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6571999999999997e-05 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791197+0000", 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8072000000000006e-05 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791302+0000", 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105414 2024-05-06T00:27:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.792824+0000", 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001521912 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795944+0000", 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031202629999999999 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796002+0000", 2024-05-06T00:27:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7547999999999999e-05 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796049+0000", 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6755999999999998e-05 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:348 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.684323423999999, 2024-05-06T00:27:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048444150000000004, 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 348 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796331+0000", 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796332+0000", 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0240000000000001e-06 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796336+0000", 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2710000000000003e-06 2024-05-06T00:27:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796348+0000", 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1843e-05 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796355+0000", 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9129999999999997e-06 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796409+0000", 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3698e-05 2024-05-06T00:27:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2955999999999997e-05 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796581+0000", 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9319000000000002e-05 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.798666+0000", 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020843089999999999 2024-05-06T00:27:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801063+0000", 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002397518 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801141+0000", 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7948000000000007e-05 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801176+0000", 2024-05-06T00:27:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4616000000000002e-05 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:351 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.684172847999999, 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049431070000000004, 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 351 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796482+0000", 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.37e-07 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796485+0000", 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.582e-06 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796495+0000", 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6350000000000001e-06 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796499+0000", 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3100000000000002e-06 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796827+0000", 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032834500000000001 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.796904+0000", 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6808000000000001e-05 2024-05-06T00:27:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797004+0000", 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9998e-05 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801185+0000", 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041808089999999997 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801284+0000", 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9228999999999995e-05 2024-05-06T00:27:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801363+0000", 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9214000000000004e-05 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.801425+0000", 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1639999999999999e-05 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:833 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [truncate 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.346393018000001, 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047565890000000003, 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 833 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134262+0000", 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2099999999999996e-07 2024-05-06T00:27:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134265+0000", 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5909999999999998e-06 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134273+0000", 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5599999999999994e-06 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134279+0000", 2024-05-06T00:27:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2580000000000003e-06 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134801+0000", 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00052267500000000001 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134865+0000", 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3507e-05 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.134939+0000", 2024-05-06T00:27:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4564e-05 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.136596+0000", 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016564749999999999 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.138912+0000", 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002316214 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.138974+0000", 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1941999999999997e-05 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.139018+0000", 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4181999999999998e-05 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.529 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1015 3.8 3:1391aa96:::rbd_data.110cc8beeb04.0000000000000025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.529 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:27:12.529 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.227173508, 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084885580000000002, 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:27:12.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253481+0000", 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253483+0000", 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0090000000000002e-06 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253489+0000", 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2639999999999997e-06 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253510+0000", 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0462000000000002e-05 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253519+0000", 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5550000000000008e-06 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253594+0000", 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4345999999999996e-05 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253675+0000", 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0860999999999997e-05 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253779+0000", 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104353 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.256496+0000", 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002716703 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261853+0000", 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053571549999999997 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261907+0000", 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4381e-05 2024-05-06T00:27:12.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261970+0000", 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2469000000000001e-05 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1017 3.7 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.22711031, 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084085759999999992, 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1017 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253544+0000", 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253545+0000", 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6000000000000003e-07 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253549+0000", 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6710000000000001e-06 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253557+0000", 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253563+0000", 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9440000000000002e-06 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253683+0000", 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120147 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253759+0000", 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5817999999999998e-05 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253858+0000", 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9053000000000002e-05 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259253+0000", 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005395047 2024-05-06T00:27:12.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261834+0000", 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025814929999999998 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261900+0000", 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5738e-05 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261953+0000", 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2784000000000001e-05 2024-05-06T00:27:12.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1018 3.7 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.227073100999998, 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085355099999999996, 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253582+0000", 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4199999999999996e-07 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253587+0000", 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7119999999999999e-06 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253595+0000", 2024-05-06T00:27:12.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7980000000000006e-06 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253601+0000", 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254070+0000", 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00046893599999999998 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254144+0000", 2024-05-06T00:27:12.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4943000000000004e-05 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254244+0000", 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9091000000000004e-05 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259286+0000", 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050428850000000004 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262019+0000", 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027324430000000002 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262072+0000", 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2994000000000001e-05 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262117+0000", 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5155999999999999e-05 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1022 3.8 3:17fe3906:::rbd_data.110cc8beeb04.0000000000000036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.227045493999999, 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085045799999999994, 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1022 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.547 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253609+0000", 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253610+0000", 2024-05-06T00:27:12.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.51e-07 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253612+0000", 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2979999999999999e-06 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253618+0000", 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.426e-06 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.253623+0000", 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7260000000000002e-06 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254047+0000", 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042410499999999998 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254100+0000", 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2592000000000001e-05 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254190+0000", 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0001999999999997e-05 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.259272+0000", 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050825840000000002 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262019+0000", 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00274654 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262071+0000", 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2091e-05 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.262114+0000", 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2664999999999999e-05 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.552 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1042 3.1e 3:7e323eed:::rbd_data.110cc8beeb04.0000000000000002:head [stat,truncate 3670016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.215551098999999, 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059170580000000002, 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1042 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265104+0000", 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265105+0000", 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.857e-06 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265112+0000", 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7769999999999997e-06 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265131+0000", 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8805e-05 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265144+0000", 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3474e-05 2024-05-06T00:27:12.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265260+0000", 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000115785 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265343+0000", 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3010000000000007e-05 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:12.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265472+0000", 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128625 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.266670+0000", 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001198382 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.270924+0000", 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042537260000000002 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.270974+0000", 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0484e-05 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271021+0000", 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6133e-05 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:1045 3.1e 3:7ff04bbe:::rbd_data.110cc8beeb04.0000000000000005:head [stat,truncate 3538944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.215481873000002, 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005939003, 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1045 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:27:12.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265173+0000", 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2500000000000005e-07 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265177+0000", 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3440000000000001e-06 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265187+0000", 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755e-05 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265194+0000", 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3999999999999997e-06 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265713+0000", 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051880999999999997 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265781+0000", 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8140999999999997e-05 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.265878+0000", 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7052999999999994e-05 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.266675+0000", 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00079742599999999995 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271049+0000", 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043737630000000001 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271077+0000", 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.775e-05 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.271112+0000", 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4935999999999999e-05 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1205 3.4 e29/18 3:26d09f8b:::rbd_object_map.110cc8beeb04:head v 29'87, mlcod=29'83)", 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.100818122, 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043375489999999996, 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.564 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379837+0000", 2024-05-06T00:27:12.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379843+0000", 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9789999999999998e-06 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379858+0000", 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4946999999999999e-05 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379865+0000", 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.754e-06 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379942+0000", 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6687000000000006e-05 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.379960+0000", 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8357999999999999e-05 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.384139+0000", 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041789239999999997 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.384174+0000", 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5221e-05 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2561 3.1e 3:7e323eed:::rbd_data.110cc8beeb04.0000000000000002:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498449+0000", 2024-05-06T00:27:12.568 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.982206034000001, 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068185010000000002, 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2561 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498449+0000", 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498449+0000", 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498451+0000", 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3860000000000001e-06 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498464+0000", 2024-05-06T00:27:12.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3477e-05 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498506+0000", 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1596e-05 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498565+0000", 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9212999999999997e-05 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498740+0000", 2024-05-06T00:27:12.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000174902 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.499058+0000", 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031793400000000002 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.499554+0000", 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00049580299999999996 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503646+0000", 2024-05-06T00:27:12.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040925689999999999 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503684+0000", 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7209999999999998e-05 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503815+0000", 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131142 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505267+0000", 2024-05-06T00:27:12.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001452269 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2567 3.1e 3:7ff04bbe:::rbd_data.110cc8beeb04.0000000000000005:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.981999286000001, 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091482830000000001, 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2567 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498657+0000", 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.75e-06 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498666+0000", 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3030000000000005e-06 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498690+0000", 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-05 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498707+0000", 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7326999999999999e-05 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501155+0000", 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002447884 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501468+0000", 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031281100000000002 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501983+0000", 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051484999999999996 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505388+0000", 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034055240000000001 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507550+0000", 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021622690000000001 2024-05-06T00:27:12.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507694+0000", 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014377899999999999 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507804+0000", 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109486 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.579 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2582 3.7 3:e05e3ccf:::rbd_data.110cc8beeb04.000000000000003b:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.977815711, 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062205380000000003, 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2582 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502841+0000", 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.497e-06 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502852+0000", 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0246e-05 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502884+0000", 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2722999999999998e-05 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502924+0000", 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9657000000000003e-05 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503194+0000", 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026947100000000002 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503406+0000", 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021234499999999999 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503867+0000", 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00046143300000000003 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508399+0000", 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045314639999999998 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508816+0000", 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041726699999999997 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508956+0000", 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000139849 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509059+0000", 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010358599999999999 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2583 3.8 3:1592c298:::rbd_data.110cc8beeb04.000000000000003c:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.503003+0000", 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.977651949, 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067545779999999998, 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2583 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503003+0000", 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503003+0000", 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503005+0000", 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9400000000000001e-06 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503029+0000", 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4695999999999999e-05 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503057+0000", 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8017e-05 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503076+0000", 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8516999999999999e-05 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503228+0000", 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015184500000000001 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503365+0000", 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013740900000000001 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503789+0000", 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042415600000000002 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508615+0000", 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048255570000000003 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:12.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509494+0000", 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087899899999999999 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509642+0000", 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000148283 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509757+0000", 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000115159 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:2578 3.9 e32/18 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head v 32'23, mlcod=29'20)", 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.503523+0000", 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.97713212, 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073103200000000004, 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503523+0000", 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503523+0000", 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503523+0000", 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.23e-07 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503535+0000", 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1297e-05 2024-05-06T00:27:12.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503557+0000", 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2532999999999999e-05 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503577+0000", 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0264000000000001e-05 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503844+0000", 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000266171 2024-05-06T00:27:12.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503892+0000", 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8192999999999998e-05 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510706+0000", 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068146730000000003 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510833+0000", 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012636599999999999 2024-05-06T00:27:12.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:12.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:12.595 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:12.595 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:12.481+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:12.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:12.597 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:12.597 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:12.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:12.600 INFO:teuthology.orchestra.run.smithi186.stdout:1064 read 0xbc7c00 thru 0xbd42ec (0xc6ed bytes) 2024-05-06T00:27:12.600 INFO:teuthology.orchestra.run.smithi186.stdout:1065 write 0x149da00 thru 0x14a92df (0xb8e0 bytes) 2024-05-06T00:27:12.600 INFO:teuthology.orchestra.run.smithi186.stdout:1066 punch from 0x15a8400 to 0x15b0a00, (0x8600 bytes) 2024-05-06T00:27:12.600 INFO:teuthology.orchestra.run.smithi186.stdout:1067 punch from 0x5eb4800 to 0x5ec3600, (0xee00 bytes) 2024-05-06T00:27:12.600 INFO:teuthology.orchestra.run.smithi186.stdout:1068 trunc from 0x70bba00 to 0x17ce800 2024-05-06T00:27:12.601 INFO:teuthology.orchestra.run.smithi186.stdout:1071 punch from 0xdc0600 to 0xdcc800, (0xc200 bytes) 2024-05-06T00:27:12.601 INFO:teuthology.orchestra.run.smithi186.stdout:1072 write 0x7da1600 thru 0x7dafc10 (0xe611 bytes) 2024-05-06T00:27:12.608 INFO:teuthology.orchestra.run.smithi186.stdout:1073 write 0x9d26600 thru 0x9d30111 (0x9b12 bytes) 2024-05-06T00:27:12.609 INFO:teuthology.orchestra.run.smithi186.stdout:1074 punch from 0x63dfa00 to 0x63e5800, (0x5e00 bytes) 2024-05-06T00:27:12.609 INFO:teuthology.orchestra.run.smithi186.stdout:1075 read 0x139b200 thru 0x139cc49 (0x1a4a bytes) 2024-05-06T00:27:12.611 INFO:teuthology.orchestra.run.smithi186.stdout:1079 trunc from 0x9d30200 to 0x92f5000 2024-05-06T00:27:12.636 INFO:teuthology.orchestra.run.smithi186.stdout:1080 read 0x4ca2000 thru 0x4ca78cd (0x58ce bytes) 2024-05-06T00:27:12.636 INFO:teuthology.orchestra.run.smithi186.stdout:1082 read 0xc89200 thru 0xc928ca (0x96cb bytes) 2024-05-06T00:27:12.637 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:12.637+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:12.639 INFO:teuthology.orchestra.run.smithi186.stdout:1083 trunc from 0x92f5000 to 0xa443400 2024-05-06T00:27:12.652 INFO:teuthology.orchestra.run.smithi186.stdout:1084 write 0x26bc000 thru 0x26c7be7 (0xbbe8 bytes) 2024-05-06T00:27:12.654 INFO:teuthology.orchestra.run.smithi186.stdout:1085 punch from 0x21f9600 to 0x2202200, (0x8c00 bytes) 2024-05-06T00:27:12.654 INFO:teuthology.orchestra.run.smithi186.stdout:1087 write 0x5414600 thru 0x541c246 (0x7c47 bytes) 2024-05-06T00:27:12.654 INFO:teuthology.orchestra.run.smithi186.stdout:1088 read 0x1a01400 thru 0x1a08634 (0x7235 bytes) 2024-05-06T00:27:12.655 INFO:teuthology.orchestra.run.smithi186.stdout:1089 read 0x619c000 thru 0x61a8ee4 (0xcee5 bytes) 2024-05-06T00:27:12.655 INFO:teuthology.orchestra.run.smithi186.stdout:1090 read 0x86ece00 thru 0x86f1129 (0x432a bytes) 2024-05-06T00:27:12.655 INFO:teuthology.orchestra.run.smithi186.stdout:1094 read 0x5504800 thru 0x550bcfb (0x74fc bytes) 2024-05-06T00:27:12.655 INFO:teuthology.orchestra.run.smithi186.stdout:1095 read 0x4cd3e00 thru 0x4ce29ad (0xebae bytes) 2024-05-06T00:27:12.656 INFO:teuthology.orchestra.run.smithi186.stdout:1097 read 0x52000 thru 0x5e040 (0xc041 bytes) 2024-05-06T00:27:12.656 INFO:teuthology.orchestra.run.smithi186.stdout:1101 punch from 0x7267800 to 0x7269c00, (0x2400 bytes) 2024-05-06T00:27:12.667 INFO:teuthology.orchestra.run.smithi186.stdout:1103 trunc from 0xa443400 to 0xe62b000 2024-05-06T00:27:12.682 INFO:teuthology.orchestra.run.smithi186.stdout:1104 read 0x3eba600 thru 0x3ebc26a (0x1c6b bytes) 2024-05-06T00:27:12.683 INFO:teuthology.orchestra.run.smithi186.stdout:1110 write 0xc9cb200 thru 0xc9ce046 (0x2e47 bytes) 2024-05-06T00:27:12.683 INFO:teuthology.orchestra.run.smithi186.stdout:1113 write 0xabdfa00 thru 0xabe8f0a (0x950b bytes) 2024-05-06T00:27:12.683 INFO:teuthology.orchestra.run.smithi186.stdout:1114 read 0xa9ef000 thru 0xa9fd075 (0xe076 bytes) 2024-05-06T00:27:12.683 INFO:teuthology.orchestra.run.smithi186.stdout:1117 punch from 0x65cac00 to 0x65cda00, (0x2e00 bytes) 2024-05-06T00:27:12.684 INFO:teuthology.orchestra.run.smithi186.stdout:1122 read 0xc8b0000 thru 0xc8b406f (0x4070 bytes) 2024-05-06T00:27:12.684 INFO:teuthology.orchestra.run.smithi186.stdout:1123 write 0x5d2c00 thru 0x5d55cf (0x29d0 bytes) 2024-05-06T00:27:12.684 INFO:teuthology.orchestra.run.smithi186.stdout:1127 write 0x12e8600 thru 0x12f0639 (0x803a bytes) 2024-05-06T00:27:12.684 INFO:teuthology.orchestra.run.smithi186.stdout:1130 write 0xbfc1800 thru 0xbfd00d6 (0xe8d7 bytes) 2024-05-06T00:27:12.684 INFO:teuthology.orchestra.run.smithi186.stdout:1131 punch from 0x8524600 to 0x8528a00, (0x4400 bytes) 2024-05-06T00:27:12.685 INFO:teuthology.orchestra.run.smithi186.stdout:1132 write 0x799a600 thru 0x79a4948 (0xa349 bytes) 2024-05-06T00:27:12.686 INFO:teuthology.orchestra.run.smithi186.stdout:1133 read 0x74a3000 thru 0x74aae7e (0x7e7f bytes) 2024-05-06T00:27:12.686 INFO:teuthology.orchestra.run.smithi186.stdout:1134 read 0x47a5800 thru 0x47b0648 (0xae49 bytes) 2024-05-06T00:27:12.686 INFO:teuthology.orchestra.run.smithi186.stdout:1135 write 0x7f3ce00 thru 0x7f466c9 (0x98ca bytes) 2024-05-06T00:27:12.690 INFO:teuthology.orchestra.run.smithi186.stdout:1137 trunc from 0xe62b000 to 0xc62f000 2024-05-06T00:27:12.700 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:12.697+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:12.720 INFO:teuthology.orchestra.run.smithi186.stdout:1138 punch from 0x213c00 to 0x21ec00, (0xb000 bytes) 2024-05-06T00:27:12.722 INFO:teuthology.orchestra.run.smithi186.stdout:1140 read 0x221f400 thru 0x222e79d (0xf39e bytes) 2024-05-06T00:27:12.723 INFO:teuthology.orchestra.run.smithi186.stdout:1141 read 0x9b0f200 thru 0x9b13352 (0x4153 bytes) 2024-05-06T00:27:12.723 INFO:teuthology.orchestra.run.smithi186.stdout:1142 read 0x9409400 thru 0x940ad08 (0x1909 bytes) 2024-05-06T00:27:12.724 INFO:teuthology.orchestra.run.smithi186.stdout:1143 read 0x1683a00 thru 0x16894b4 (0x5ab5 bytes) 2024-05-06T00:27:12.724 INFO:teuthology.orchestra.run.smithi186.stdout:1144 read 0x3115600 thru 0x3121e05 (0xc806 bytes) 2024-05-06T00:27:12.724 INFO:teuthology.orchestra.run.smithi186.stdout:1146 write 0xa54f800 thru 0xa555db9 (0x65ba bytes) 2024-05-06T00:27:12.724 INFO:teuthology.orchestra.run.smithi186.stdout:1147 read 0x94dd000 thru 0x94dd66d (0x66e bytes) 2024-05-06T00:27:12.734 INFO:teuthology.orchestra.run.smithi186.stdout:1148 trunc from 0xc62f000 to 0x71ca600 2024-05-06T00:27:12.743 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:12.743 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:12.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.743 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:3717 3.a 3:57bf73b9:::rbd_header.110cd55b09e5:head [call lock.lock in=76b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:12.743 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:12.744719+0000", 2024-05-06T00:27:12.743 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.00059290000000000005, 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062062399999999996, 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3717 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:12.744719+0000", 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:12.744719+0000", 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:12.744721+0000", 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:12.744732+0000", 2024-05-06T00:27:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0203e-05 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:12.744760+0000", 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8751999999999998e-05 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:12.744777+0000", 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6466000000000001e-05 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:12.744871+0000", 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4271999999999995e-05 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:12.745042+0000", 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017038599999999999 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-05-06T00:27:12.747 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:12.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:12.771 INFO:teuthology.orchestra.run.smithi186.stdout:1149 write 0xdbaa00 thru 0xdc7ba6 (0xd1a7 bytes) 2024-05-06T00:27:12.774 INFO:teuthology.orchestra.run.smithi186.stdout:1151 read 0x468a000 thru 0x46996e4 (0xf6e5 bytes) 2024-05-06T00:27:12.774 INFO:teuthology.orchestra.run.smithi186.stdout:1152 write 0xc45200 thru 0xc53949 (0xe74a bytes) 2024-05-06T00:27:12.774 INFO:teuthology.orchestra.run.smithi186.stdout:1154 read 0x5d9ca00 thru 0x5da5690 (0x8c91 bytes) 2024-05-06T00:27:12.774 INFO:teuthology.orchestra.run.smithi186.stdout:1155 punch from 0x4362a00 to 0x436b400, (0x8a00 bytes) 2024-05-06T00:27:12.775 INFO:teuthology.orchestra.run.smithi186.stdout:1158 write 0x13bc200 thru 0x13c69ff (0xa800 bytes) 2024-05-06T00:27:12.786 INFO:teuthology.orchestra.run.smithi186.stdout:1159 trunc from 0x71ca600 to 0xdb0be00 2024-05-06T00:27:12.801 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:12.797+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:12.810 INFO:teuthology.orchestra.run.smithi186.stdout:1161 write 0xccd1800 thru 0xccd6e4d (0x564e bytes) 2024-05-06T00:27:12.812 INFO:teuthology.orchestra.run.smithi186.stdout:1163 write 0x4d7b600 thru 0x4d8876e (0xd16f bytes) 2024-05-06T00:27:12.812 INFO:teuthology.orchestra.run.smithi186.stdout:1164 read 0x6ddba00 thru 0x6de8abc (0xd0bd bytes) 2024-05-06T00:27:12.812 INFO:teuthology.orchestra.run.smithi186.stdout:1165 read 0x7ee4400 thru 0x7eed593 (0x9194 bytes) 2024-05-06T00:27:12.813 INFO:teuthology.orchestra.run.smithi186.stdout:1166 write 0xaa9de00 thru 0xaaa6a8b (0x8c8c bytes) 2024-05-06T00:27:12.813 INFO:teuthology.orchestra.run.smithi186.stdout:1168 punch from 0x91e6600 to 0x91eae00, (0x4800 bytes) 2024-05-06T00:27:12.813 INFO:teuthology.orchestra.run.smithi186.stdout:1172 trunc from 0xdb0be00 to 0xda2cc00 2024-05-06T00:27:12.857 INFO:teuthology.orchestra.run.smithi186.stdout:1174 trunc from 0xda2cc00 to 0x90e1000 2024-05-06T00:27:12.883 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:12.883 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:12.898 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:27:12.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:12.902 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:12.900+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:12.915 INFO:teuthology.orchestra.run.smithi186.stdout:1177 trunc from 0x90e1000 to 0x38e8a00 2024-05-06T00:27:12.943 INFO:teuthology.orchestra.run.smithi186.stdout:1179 write 0x35ae400 thru 0x35aeea7 (0xaa8 bytes) 2024-05-06T00:27:12.966 INFO:teuthology.orchestra.run.smithi186.stdout:1180 write 0x3dc4800 thru 0x3dd0e6b (0xc66c bytes) 2024-05-06T00:27:12.967 INFO:teuthology.orchestra.run.smithi186.stdout:1182 read 0x2b3f600 thru 0x2b4c9a0 (0xd3a1 bytes) 2024-05-06T00:27:12.970 INFO:teuthology.orchestra.run.smithi186.stdout:1183 trunc from 0x3dd1000 to 0x53e8a00 2024-05-06T00:27:12.991 INFO:teuthology.orchestra.run.smithi186.stdout:1184 trunc from 0x53e8a00 to 0x4734e00 2024-05-06T00:27:13.002 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:13.000+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:13.013 INFO:teuthology.orchestra.run.smithi186.stdout:1186 trunc from 0x4734e00 to 0x497c800 2024-05-06T00:27:13.035 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:13.035 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:13.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.432588213999999, 2024-05-06T00:27:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.653758429, 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012319788, 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381229+0000", 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381231+0000", 2024-05-06T00:27:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.624e-06 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381243+0000", 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2476e-05 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381263+0000", 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-05 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381278+0000", 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5183e-05 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389490+0000", 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082113770000000006 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389614+0000", 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012416900000000001 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.389813+0000", 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199321 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391953+0000", 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021402869999999998 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393301+0000", 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001347729 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393438+0000", 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136666 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393549+0000", 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110856 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.653682625999998, 2024-05-06T00:27:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014641942999999999, 2024-05-06T00:27:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46 2024-05-06T00:27:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381305+0000", 2024-05-06T00:27:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381306+0000", 2024-05-06T00:27:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.403e-06 2024-05-06T00:27:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381316+0000", 2024-05-06T00:27:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-05-06T00:27:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381341+0000", 2024-05-06T00:27:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4372999999999999e-05 2024-05-06T00:27:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381356+0000", 2024-05-06T00:27:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5207e-05 2024-05-06T00:27:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390708+0000", 2024-05-06T00:27:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093519799999999993 2024-05-06T00:27:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.390856+0000", 2024-05-06T00:27:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839200000000001 2024-05-06T00:27:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.391064+0000", 2024-05-06T00:27:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000207302 2024-05-06T00:27:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394636+0000", 2024-05-06T00:27:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003572563 2024-05-06T00:27:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395813+0000", 2024-05-06T00:27:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011765910000000001 2024-05-06T00:27:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395885+0000", 2024-05-06T00:27:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2601999999999994e-05 2024-05-06T00:27:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395947+0000", 2024-05-06T00:27:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1183e-05 2024-05-06T00:27:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.653605949999999, 2024-05-06T00:27:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017147241000000001, 2024-05-06T00:27:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 47 2024-05-06T00:27:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381381+0000", 2024-05-06T00:27:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381383+0000", 2024-05-06T00:27:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6199999999999999e-06 2024-05-06T00:27:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381504+0000", 2024-05-06T00:27:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120709 2024-05-06T00:27:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381525+0000", 2024-05-06T00:27:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1705999999999999e-05 2024-05-06T00:27:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381541+0000", 2024-05-06T00:27:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5367e-05 2024-05-06T00:27:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392051+0000", 2024-05-06T00:27:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010510442 2024-05-06T00:27:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392177+0000", 2024-05-06T00:27:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012575600000000001 2024-05-06T00:27:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.392387+0000", 2024-05-06T00:27:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020954799999999999 2024-05-06T00:27:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394744+0000", 2024-05-06T00:27:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023578319999999998 2024-05-06T00:27:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398377+0000", 2024-05-06T00:27:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036325189999999999 2024-05-06T00:27:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398462+0000", 2024-05-06T00:27:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4668999999999999e-05 2024-05-06T00:27:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398529+0000", 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7073e-05 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.653405935000002, 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017264267, 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 48 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381581+0000", 2024-05-06T00:27:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381583+0000", 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7069999999999999e-06 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381594+0000", 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1365999999999999e-05 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381618+0000", 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4014000000000001e-05 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381639+0000", 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0616e-05 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393664+0000", 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012025179 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393804+0000", 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014017599999999999 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.393998+0000", 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019390499999999999 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398573+0000", 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045742789999999997 2024-05-06T00:27:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398683+0000", 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110136 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398786+0000", 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103259 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398846+0000", 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9630000000000003e-05 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.653314672, 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018281285000000001, 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 49 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381673+0000", 2024-05-06T00:27:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381674+0000", 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.559e-06 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381685+0000", 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1049999999999999e-05 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381705+0000", 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0227000000000001e-05 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381728+0000", 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2467e-05 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394829+0000", 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013101358 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.394953+0000", 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012340499999999999 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.395145+0000", 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019252900000000001 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399714+0000", 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045683620000000003 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399788+0000", 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4719000000000001e-05 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399871+0000", 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3092999999999999e-05 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399954+0000", 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2515999999999994e-05 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.653224993000002, 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020991408, 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 50 2024-05-06T00:27:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381762+0000", 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381764+0000", 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381894+0000", 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012972600000000001 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381927+0000", 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3399000000000003e-05 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381938+0000", 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0677e-05 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396032+0000", 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014094257000000001 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396199+0000", 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016722899999999999 2024-05-06T00:27:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.396415+0000", 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021597200000000001 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.400244+0000", 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038293009999999998 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402502+0000", 2024-05-06T00:27:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002257972 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402645+0000", 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014225199999999999 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402754+0000", 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010917300000000001 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.653027618000003, 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021178922999999999, 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 51 2024-05-06T00:27:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381960+0000", 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381961+0000", 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381968+0000", 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.007e-06 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381979+0000", 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1197e-05 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.381987+0000", 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9629999999999998e-06 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397517+0000", 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015529589999999999 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397635+0000", 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011846899999999999 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.397830+0000", 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019498200000000001 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402867+0000", 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050367240000000002 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403002+0000", 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134609 2024-05-06T00:27:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403082+0000", 2024-05-06T00:27:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0666000000000003e-05 2024-05-06T00:27:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403139+0000", 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6462999999999998e-05 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.652986861000002, 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021264655, 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382000+0000", 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382001+0000", 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7300000000000005e-07 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382007+0000", 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000004e-06 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382017+0000", 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9799999999999993e-06 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.382029+0000", 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1420000000000001e-05 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.398910+0000", 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016881075999999998 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399069+0000", 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015880199999999999 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.399327+0000", 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025800200000000001 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.402970+0000", 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003643725 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403155+0000", 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018459100000000001 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403221+0000", 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6106999999999996e-05 2024-05-06T00:27:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.403265+0000", 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4135999999999999e-05 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.449389543000002, 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012456859000000001, 2024-05-06T00:27:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585598+0000", 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585599+0000", 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5899999999999995e-07 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585692+0000", 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3398000000000003e-05 2024-05-06T00:27:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585704+0000", 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2318e-05 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585713+0000", 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7649999999999999e-06 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593668+0000", 2024-05-06T00:27:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079544980000000008 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593723+0000", 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5914e-05 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.593815+0000", 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1583999999999996e-05 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595554+0000", 2024-05-06T00:27:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017386190000000001 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.597968+0000", 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024139500000000002 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598008+0000", 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9962000000000001e-05 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598055+0000", 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7092000000000002e-05 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.449257285000002, 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013009979, 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 76 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585730+0000", 2024-05-06T00:27:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585731+0000", 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0999999999999997e-07 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585738+0000", 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6059999999999999e-06 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585752+0000", 2024-05-06T00:27:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4248e-05 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.585762+0000", 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8660000000000001e-06 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594480+0000", 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087185100000000005 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594616+0000", 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135375 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.594806+0000", 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019038099999999999 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.595986+0000", 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011804039999999999 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598676+0000", 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026891889999999998 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598710+0000", 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4471999999999998e-05 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.598740+0000", 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0018e-05 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:401 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 786432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225771043000002, 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012526862999999999, 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 401 2024-05-06T00:27:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809216+0000", 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809218+0000", 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809224+0000", 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.037e-06 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809242+0000", 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809256+0000", 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4027e-05 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821381+0000", 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012125268999999999 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821510+0000", 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129219 2024-05-06T00:27:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821743+0000", 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023277500000000001 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:402 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 1703936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225709089999999, 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013133265, 2024-05-06T00:27:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 402 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809278+0000", 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809279+0000", 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809285+0000", 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:27:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809303+0000", 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7983e-05 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809317+0000", 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3856000000000001e-05 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.821841+0000", 2024-05-06T00:27:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012524400999999999 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822066+0000", 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022526100000000001 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822412+0000", 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034507400000000001 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:403 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 2621440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225653007999998, 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013546829999999999, 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 403 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809334+0000", 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809335+0000", 2024-05-06T00:27:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1200000000000001e-06 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809341+0000", 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9200000000000001e-06 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809360+0000", 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8184000000000001e-05 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809373+0000", 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822550+0000", 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013176998000000001 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822703+0000", 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015292600000000001 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822881+0000", 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017814600000000001 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:404 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 3538944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225553578, 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013816116999999999, 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 404 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809434+0000", 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809435+0000", 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4300000000000001e-06 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809441+0000", 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9730000000000002e-06 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809461+0000", 2024-05-06T00:27:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9786e-05 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809498+0000", 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6706999999999999e-05 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.822957+0000", 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013459296000000001 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823087+0000", 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054200000000001 2024-05-06T00:27:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823250+0000", 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016238299999999999 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:405 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 4456448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225467914999999, 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014023473999999999, 2024-05-06T00:27:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 405 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809519+0000", 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809521+0000", 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3629999999999999e-06 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809527+0000", 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1369999999999999e-06 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809544+0000", 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7516e-05 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809559+0000", 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.455e-05 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823295+0000", 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013736474 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823397+0000", 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010108899999999999 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823543+0000", 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014634500000000001 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:406 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 5373952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225409493000001, 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01426091, 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 406 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809578+0000", 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809579+0000", 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.077e-06 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809584+0000", 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3530000000000002e-06 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809604+0000", 2024-05-06T00:27:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9962999999999998e-05 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809632+0000", 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7710000000000001e-05 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823589+0000", 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013957476 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823695+0000", 2024-05-06T00:27:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010535900000000001 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823839+0000", 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143972 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:407 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 6291456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225335456, 2024-05-06T00:27:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014464551000000001, 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 407 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809652+0000", 2024-05-06T00:27:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809653+0000", 2024-05-06T00:27:13.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2330000000000001e-06 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809659+0000", 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0170000000000002e-06 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809678+0000", 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8703e-05 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809692+0000", 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4326e-05 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823880+0000", 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014187936999999999 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.823982+0000", 2024-05-06T00:27:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102266 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824116+0000", 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013406900000000001 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:408 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225259427000001, 2024-05-06T00:27:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014667517, 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 408 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809728+0000", 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809729+0000", 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5430000000000001e-06 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809749+0000", 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9227000000000001e-05 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809769+0000", 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0556000000000001e-05 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809784+0000", 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4507e-05 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824158+0000", 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014374418999999999 2024-05-06T00:27:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824258+0000", 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100209 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824395+0000", 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013705600000000001 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:409 3.13 3:cacb532b:::rbd_data.110c2013632c.0000000000000000:5 [sparse-read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.225183801, 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014988434, 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 409 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809804+0000", 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809805+0000", 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.917e-06 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809812+0000", 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5930000000000002e-06 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809830+0000", 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8159e-05 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.809844+0000", 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3764000000000001e-05 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824478+0000", 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014633703 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824611+0000", 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132893 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.824792+0000", 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018140499999999999 2024-05-06T00:27:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.161 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.161 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.161 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:13.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:13.165 INFO:teuthology.orchestra.run.smithi186.stdout:1187 write 0x8368c00 thru 0x8371580 (0x8981 bytes) 2024-05-06T00:27:13.165 INFO:teuthology.orchestra.run.smithi186.stdout:1188 punch from 0x242200 to 0x246e00, (0x4c00 bytes) 2024-05-06T00:27:13.165 INFO:teuthology.orchestra.run.smithi186.stdout:1189 write 0xb983000 thru 0xb9834ab (0x4ac bytes) 2024-05-06T00:27:13.165 INFO:teuthology.orchestra.run.smithi186.stdout:1191 write 0x9cc9e00 thru 0x9cd9a0e (0xfc0f bytes) 2024-05-06T00:27:13.165 INFO:teuthology.orchestra.run.smithi186.stdout:1193 write 0xc7e2e00 thru 0xc7e502f (0x2230 bytes) 2024-05-06T00:27:13.165 INFO:teuthology.orchestra.run.smithi186.stdout:1199 read 0x6693000 thru 0x669a3ea (0x73eb bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1200 read 0x56b0c00 thru 0x56bf68c (0xea8d bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1201 punch from 0x2b0d800 to 0x2b0de00, (0x600 bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1202 read 0x67ee000 thru 0x67f8e3d (0xae3e bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1203 write 0xa33e600 thru 0xa34cb14 (0xe515 bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1205 read 0x50de000 thru 0x50ea6fa (0xc6fb bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1207 punch from 0xa4a0200 to 0xa4a9a00, (0x9800 bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1212 write 0x5ce2600 thru 0x5ce6fdb (0x49dc bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1213 read 0x5cd3600 thru 0x5ce1b1c (0xe51d bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1215 write 0x2466e00 thru 0x246c0fa (0x52fb bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1217 write 0x8b20000 thru 0x8b22acf (0x2ad0 bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1218 read 0xaa50e00 thru 0xaa5b960 (0xab61 bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1220 write 0x762c600 thru 0x7638dde (0xc7df bytes) 2024-05-06T00:27:13.166 INFO:teuthology.orchestra.run.smithi186.stdout:1221 read 0xabec600 thru 0xabf9d74 (0xd775 bytes) 2024-05-06T00:27:13.167 INFO:teuthology.orchestra.run.smithi186.stdout:1224 write 0x89b4600 thru 0x89bb125 (0x6b26 bytes) 2024-05-06T00:27:13.167 INFO:teuthology.orchestra.run.smithi186.stdout:1226 write 0x53f1600 thru 0x53f6c96 (0x5697 bytes) 2024-05-06T00:27:13.167 INFO:teuthology.orchestra.run.smithi186.stdout:1227 write 0xb4cfc00 thru 0xb4db196 (0xb597 bytes) 2024-05-06T00:27:13.167 INFO:teuthology.orchestra.run.smithi186.stdout:1229 trunc from 0xc7e5200 to 0x7128a00 2024-05-06T00:27:13.167 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:13.161+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:13.168 INFO:teuthology.orchestra.run.smithi186.stdout:1231 punch from 0x5915200 to 0x591dc00, (0x8a00 bytes) 2024-05-06T00:27:13.170 INFO:teuthology.orchestra.run.smithi186.stdout:1232 read 0x2c2ec00 thru 0x2c359b1 (0x6db2 bytes) 2024-05-06T00:27:13.170 INFO:teuthology.orchestra.run.smithi186.stdout:1233 read 0x6061800 thru 0x606bab7 (0xa2b8 bytes) 2024-05-06T00:27:13.170 INFO:teuthology.orchestra.run.smithi186.stdout:1234 write 0x15c4400 thru 0x15d4269 (0xfe6a bytes) 2024-05-06T00:27:13.171 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:13.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-05-06T00:27:13.172 INFO:teuthology.orchestra.run.smithi186.stdout:1235 write 0x7075c00 thru 0x707c18e (0x658f bytes) 2024-05-06T00:27:13.182 INFO:teuthology.orchestra.run.smithi186.stdout:1236 trunc from 0x7128a00 to 0xd111800 2024-05-06T00:27:13.204 INFO:teuthology.orchestra.run.smithi186.stdout:1238 read 0x83e7e00 thru 0x83f32b9 (0xb4ba bytes) 2024-05-06T00:27:13.219 INFO:teuthology.orchestra.run.smithi186.stdout:1241 trunc from 0xd111800 to 0x5410e00 2024-05-06T00:27:13.244 INFO:teuthology.orchestra.run.smithi186.stdout:1245 flatten 2024-05-06T00:27:13.262 INFO:teuthology.orchestra.run.smithi186.stdout:1246 punch from 0xdbe200 to 0xdbe600, (0x400 bytes) 2024-05-06T00:27:13.263 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:13.260+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:13.263 INFO:teuthology.orchestra.run.smithi186.stdout:1247 write 0x3797a00 thru 0x37a4b4d (0xd14e bytes) 2024-05-06T00:27:13.272 INFO:teuthology.orchestra.run.smithi186.stdout:1248 trunc from 0x5410e00 to 0xb7ae00 2024-05-06T00:27:13.332 INFO:teuthology.orchestra.run.smithi186.stdout:1250 write 0x5a30c00 thru 0x5a33c74 (0x3075 bytes) 2024-05-06T00:27:13.334 INFO:teuthology.orchestra.run.smithi186.stdout:1251 read 0x3990600 thru 0x399d2fd (0xccfe bytes) 2024-05-06T00:27:13.334 INFO:teuthology.orchestra.run.smithi186.stdout:1256 read 0x70f600 thru 0x714ac6 (0x54c7 bytes) 2024-05-06T00:27:13.342 INFO:teuthology.orchestra.run.smithi186.stdout:1261 trunc from 0x5a33e00 to 0x8d88a00 2024-05-06T00:27:13.358 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:13.359 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:13.359 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:13.359 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:13.364 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:13.361+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:13.365 INFO:teuthology.orchestra.run.smithi186.stdout:1262 write 0x1a53800 thru 0x1a5567f (0x1e80 bytes) 2024-05-06T00:27:13.366 INFO:teuthology.orchestra.run.smithi186.stdout:1265 read 0x1a6a800 thru 0x1a737fe (0x8fff bytes) 2024-05-06T00:27:13.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:13.374 INFO:teuthology.orchestra.run.smithi186.stdout:1273 trunc from 0x8d88a00 to 0xbc44400 2024-05-06T00:27:13.395 INFO:teuthology.orchestra.run.smithi186.stdout:1274 write 0x34c9c00 thru 0x34d2534 (0x8935 bytes) 2024-05-06T00:27:13.397 INFO:teuthology.orchestra.run.smithi186.stdout:1275 punch from 0x6ca5200 to 0x6cae600, (0x9400 bytes) 2024-05-06T00:27:13.398 INFO:teuthology.orchestra.run.smithi186.stdout:1276 punch from 0x4247c00 to 0x4254800, (0xcc00 bytes) 2024-05-06T00:27:13.398 INFO:teuthology.orchestra.run.smithi186.stdout:1277 punch from 0x148e000 to 0x1499e00, (0xbe00 bytes) 2024-05-06T00:27:13.413 INFO:teuthology.orchestra.run.smithi186.stdout:1278 trunc from 0xbc44400 to 0x4352800 2024-05-06T00:27:13.440 INFO:teuthology.orchestra.run.smithi186.stdout:1279 read 0x2030e00 thru 0x203f9fa (0xebfb bytes) 2024-05-06T00:27:13.442 INFO:teuthology.orchestra.run.smithi186.stdout:1280 write 0x5eee00 thru 0x5fe4aa (0xf6ab bytes) 2024-05-06T00:27:13.443 INFO:teuthology.orchestra.run.smithi186.stdout:1281 read 0xe04400 thru 0xe05f13 (0x1b14 bytes) 2024-05-06T00:27:13.443 INFO:teuthology.orchestra.run.smithi186.stdout:1282 punch from 0x37f4c00 to 0x3801400, (0xc800 bytes) 2024-05-06T00:27:13.444 INFO:teuthology.orchestra.run.smithi186.stdout:1284 read 0x1948e00 thru 0x195547e (0xc67f bytes) 2024-05-06T00:27:13.444 INFO:teuthology.orchestra.run.smithi186.stdout:1285 read 0x113fe00 thru 0x1148e0e (0x900f bytes) 2024-05-06T00:27:13.465 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:13.465+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:13.465 INFO:teuthology.orchestra.run.smithi186.stdout:1286 write 0x4547200 thru 0x4554500 (0xd301 bytes) 2024-05-06T00:27:13.467 INFO:teuthology.orchestra.run.smithi186.stdout:1287 punch from 0x21c5e00 to 0x21d5600, (0xf800 bytes) 2024-05-06T00:27:13.468 INFO:teuthology.orchestra.run.smithi186.stdout:1288 punch from 0x2edf600 to 0x2ee3200, (0x3c00 bytes) 2024-05-06T00:27:13.478 INFO:teuthology.orchestra.run.smithi186.stdout:1290 trunc from 0x4554600 to 0x97ba800 2024-05-06T00:27:13.494 INFO:teuthology.orchestra.run.smithi186.stdout:1291 read 0x203c600 thru 0x2049fef (0xd9f0 bytes) 2024-05-06T00:27:13.495 INFO:teuthology.orchestra.run.smithi186.stdout:1293 read 0x25ff600 thru 0x2600e96 (0x1897 bytes) 2024-05-06T00:27:13.496 INFO:teuthology.orchestra.run.smithi186.stdout:1296 write 0x5a45a00 thru 0x5a4c1b9 (0x67ba bytes) 2024-05-06T00:27:13.496 INFO:teuthology.orchestra.run.smithi186.stdout:1298 write 0x8e93000 thru 0x8e9c90f (0x9910 bytes) 2024-05-06T00:27:13.496 INFO:teuthology.orchestra.run.smithi186.stdout:1299 write 0x91b7400 thru 0x91bc33e (0x4f3f bytes) 2024-05-06T00:27:13.529 INFO:teuthology.orchestra.run.smithi186.stdout:1300 write 0xe822400 thru 0xe82f8b9 (0xd4ba bytes) 2024-05-06T00:27:13.548 INFO:teuthology.orchestra.run.smithi186.stdout:1302 trunc from 0xe82fa00 to 0x4757200 2024-05-06T00:27:13.550 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:13.550 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:13.550 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:13.550 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:13.551 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:13.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:13.567 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:13.565+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:13.583 INFO:teuthology.orchestra.run.smithi186.stdout:1304 punch from 0x11a0e00 to 0x11ada00, (0xcc00 bytes) 2024-05-06T00:27:13.585 INFO:teuthology.orchestra.run.smithi186.stdout:1305 read 0x1ce3800 thru 0x1ce9ff4 (0x67f5 bytes) 2024-05-06T00:27:13.585 INFO:teuthology.orchestra.run.smithi186.stdout:1307 read 0xc88000 thru 0xc8cdb6 (0x4db7 bytes) 2024-05-06T00:27:13.612 INFO:teuthology.orchestra.run.smithi186.stdout:1308 write 0xb9eee00 thru 0xb9fa40b (0xb60c bytes) 2024-05-06T00:27:13.631 INFO:teuthology.orchestra.run.smithi186.stdout:1309 trunc from 0xb9fa600 to 0x1238800 2024-05-06T00:27:13.667 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:13.664+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:13.677 INFO:teuthology.orchestra.run.smithi186.stdout:1310 punch from 0xe53000 to 0xe60e00, (0xde00 bytes) 2024-05-06T00:27:13.679 INFO:teuthology.orchestra.run.smithi186.stdout:1314 punch from 0x5ef600 to 0x5f4e00, (0x5800 bytes) 2024-05-06T00:27:13.680 INFO:teuthology.orchestra.run.smithi186.stdout:1315 clone 9 order 18 su 65536 sc 11 2024-05-06T00:27:13.706 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:13.706 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:13.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:13.706 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:13.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.706 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:9 3.1a 3:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-06T00:27:13.706 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.989182988, 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084089950000000007, 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 9 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715198+0000", 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715203+0000", 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715219+0000", 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6577000000000001e-05 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715254+0000", 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5308999999999997e-05 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715279+0000", 2024-05-06T00:27:13.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4902999999999999e-05 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715445+0000", 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016536900000000001 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.715456+0000", 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1037e-05 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.719765+0000", 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043095060000000003 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.720002+0000", 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000236596 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.720369+0000", 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036678999999999998 2024-05-06T00:27:13.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.722404+0000", 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002035121 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723446+0000", 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010426249999999999 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723534+0000", 2024-05-06T00:27:13.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7423000000000006e-05 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:45.723607+0000", 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3195999999999995e-05 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4412.0:632 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:13.713 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.966655452000001, 2024-05-06T00:27:13.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073037140000000002, 2024-05-06T00:27:13.714 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.714 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.714 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4412", 2024-05-06T00:27:13.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 632 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737726+0000", 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737728+0000", 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6699999999999998e-06 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737738+0000", 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0263e-05 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737764+0000", 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5584000000000001e-05 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737781+0000", 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6849000000000001e-05 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737901+0000", 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012037299999999999 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.737912+0000", 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0343000000000001e-05 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.744738+0000", 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068260940000000004 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.744863+0000", 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125139 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745029+0000", 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016639899999999999 2024-05-06T00:27:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:342 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.964407496, 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053952979999999998, 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:27:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 342 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739973+0000", 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739976+0000", 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.88e-06 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.739988+0000", 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1440000000000001e-05 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740019+0000", 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0942999999999998e-05 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740034+0000", 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4766000000000001e-05 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740219+0000", 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185069 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.740229+0000", 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0890000000000001e-05 2024-05-06T00:27:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745083+0000", 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048533760000000004 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745208+0000", 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012539499999999999 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:49.745369+0000", 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016053899999999999 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:303 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2401792~22528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.933796306000001, 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057218729999999997, 2024-05-06T00:27:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 303 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770585+0000", 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770589+0000", 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9099999999999998e-06 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770633+0000", 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4326999999999998e-05 2024-05-06T00:27:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770670+0000", 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7101999999999998e-05 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770692+0000", 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1512999999999999e-05 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.770851+0000", 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015903600000000001 2024-05-06T00:27:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.771219+0000", 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0003681 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.771608+0000", 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038980699999999999 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.773584+0000", 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019750890000000002 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776110+0000", 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025268669999999999 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776236+0000", 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125726 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.776307+0000", 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0395999999999995e-05 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:325 3.9 3:90865cbe:::rbd_data.110cc8beeb04.0000000000000015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.911494368, 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0077305209999999997, 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 325 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792887+0000", 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792891+0000", 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9269999999999998e-06 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792905+0000", 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4926000000000001e-05 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.792989+0000", 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3999999999999995e-05 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793013+0000", 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3856e-05 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795543+0000", 2024-05-06T00:27:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025296369999999999 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.795768+0000", 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022517500000000001 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.796053+0000", 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028487500000000003 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798879+0000", 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028255210000000001 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800464+0000", 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001585218 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800548+0000", 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4635999999999994e-05 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.800617+0000", 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8750000000000004e-05 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:324 3.19 e29/18 3:99a6ca97:::rbd_data.110cc8beeb04.0000000000000013:head v 29'64, mlcod=28'62)", 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.911242473000001, 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052824120000000002, 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793139+0000", 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793140+0000", 2024-05-06T00:27:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3799999999999999e-06 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793152+0000", 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2046e-05 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793186+0000", 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3646999999999998e-05 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.793202+0000", 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6220000000000001e-05 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794514+0000", 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001312353 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.794578+0000", 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3476e-05 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798323+0000", 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037458109999999999 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.798421+0000", 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7479000000000006e-05 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:343 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.907304740000001, 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062599750000000001, 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 343 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797076+0000", 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797078+0000", 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0700000000000001e-06 2024-05-06T00:27:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797088+0000", 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6630000000000005e-06 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797124+0000", 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6186000000000002e-05 2024-05-06T00:27:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797147+0000", 2024-05-06T00:27:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2819999999999998e-05 2024-05-06T00:27:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797305+0000", 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015812199999999999 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797480+0000", 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000175172 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.797716+0000", 2024-05-06T00:27:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235419 2024-05-06T00:27:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.799507+0000", 2024-05-06T00:27:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001791089 2024-05-06T00:27:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803182+0000", 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003674835 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803266+0000", 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3986000000000005e-05 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.803336+0000", 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0612999999999996e-05 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2577 3.15 3:afc90aa8:::rbd_data.110cc8beeb04.0000000000000023:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.199909296, 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056316919999999998, 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2577 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504473+0000", 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2619999999999999e-06 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504479+0000", 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8080000000000001e-06 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504494+0000", 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5356999999999999e-05 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504504+0000", 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6739999999999992e-06 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504566+0000", 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2030000000000001e-05 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504653+0000", 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6718000000000006e-05 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504831+0000", 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017891900000000001 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.506697+0000", 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018657529999999999 2024-05-06T00:27:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510025+0000", 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003327909 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510063+0000", 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8243e-05 2024-05-06T00:27:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.510103+0000", 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0018999999999997e-05 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:2578 3.9 3:93967e8c:::rbd_data.110cc8beeb04.0000000000000026:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.19984884, 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082191650000000005, 2024-05-06T00:27:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2578 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504532+0000", 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504533+0000", 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8700000000000005e-07 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504539+0000", 2024-05-06T00:27:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5509999999999996e-06 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504551+0000", 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1319e-05 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504559+0000", 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5720000000000003e-06 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504632+0000", 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2531000000000001e-05 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504726+0000", 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4475000000000006e-05 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.504935+0000", 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020863699999999999 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.506757+0000", 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001822471 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512670+0000", 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059127989999999998 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512717+0000", 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6320999999999999e-05 2024-05-06T00:27:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.512751+0000", 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4702000000000001e-05 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4578.0:2576 3.a e32/18 3:5029607f:::rbd_data.110cc8beeb04.0000000000000017:head v 32'20, mlcod=29'17)", 2024-05-06T00:27:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.198883283000001, 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063083260000000004, 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505498+0000", 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505499+0000", 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0610000000000001e-06 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505508+0000", 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2310000000000002e-06 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505522+0000", 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4419999999999999e-05 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505532+0000", 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2220000000000008e-06 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505599+0000", 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7485999999999997e-05 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.505619+0000", 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9995000000000001e-05 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.511754+0000", 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061349300000000002 2024-05-06T00:27:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:00.511806+0000", 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1981000000000003e-05 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:758 3.3 3:c1d613dd:::rbd_data.110c6af4f54c.0000000000000007:head [stat,truncate 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.211489+0000", 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4928918199999996, 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048893859999999999, 2024-05-06T00:27:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 758 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.211489+0000", 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.211489+0000", 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.211492+0000", 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9699999999999999e-06 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.211506+0000", 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3482999999999999e-05 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.211544+0000", 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8000999999999998e-05 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.211564+0000", 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0665999999999998e-05 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.211824+0000", 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025970300000000002 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.212027+0000", 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020266199999999999 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.212357+0000", 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032998599999999999 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.215015+0000", 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026583470000000001 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.216185+0000", 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001170471 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.216290+0000", 2024-05-06T00:27:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104522 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.216379+0000", 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8574999999999998e-05 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:881 3.3 3:c1d613dd:::rbd_data.110c6af4f54c.0000000000000007:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.028447+0000", 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.675933626, 2024-05-06T00:27:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0069936770000000002, 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 881 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028447+0000", 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028447+0000", 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028449+0000", 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.114e-06 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028459+0000", 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2909999999999996e-06 2024-05-06T00:27:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028487+0000", 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8212999999999999e-05 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028500+0000", 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3406e-05 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028589+0000", 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8861000000000002e-05 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028699+0000", 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110185 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028963+0000", 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000263572 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.032598+0000", 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036345819999999999 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.035290+0000", 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026923810000000002 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.035374+0000", 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3738000000000004e-05 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.035441+0000", 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7334000000000003e-05 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4737.0:878 3.1 e37/18 3:832629b4:::rbd_data.110c6af4f54c.0000000000000005:head v 37'32, mlcod=36'29)", 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.029907+0000", 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6744739389999999, 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081586839999999994, 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.029907+0000", 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.029907+0000", 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.029908+0000", 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7000000000000005e-07 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.029917+0000", 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4970000000000007e-06 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030317+0000", 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040026200000000002 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030329+0000", 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1379999999999999e-05 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030406+0000", 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7336999999999995e-05 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030435+0000", 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8940000000000001e-05 2024-05-06T00:27:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037985+0000", 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075503860000000001 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038066+0000", 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0311999999999999e-05 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:888 3.15 3:adc7bda1:::rbd_data.110c6af4f54c.000000000000000a:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.029958+0000", 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6744225510000001, 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008090758, 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 888 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.029958+0000", 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.029958+0000", 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.029960+0000", 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.342e-06 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.029987+0000", 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6829e-05 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030000+0000", 2024-05-06T00:27:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3322000000000001e-05 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030011+0000", 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1242e-05 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030087+0000", 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5729000000000006e-05 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030196+0000", 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108925 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030449+0000", 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025350699999999998 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.034902+0000", 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044531429999999997 2024-05-06T00:27:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037908+0000", 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030050519999999998 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037970+0000", 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2557000000000004e-05 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038049+0000", 2024-05-06T00:27:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9110000000000007e-05 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4737.0:883 3.a e37/18 3:560873e5:::rbd_data.110c6af4f54c.0000000000000008:head v 37'51, mlcod=37'48)", 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.030048+0000", 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.674333029, 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0078101459999999996, 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030048+0000", 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030048+0000", 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030049+0000", 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:27:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030056+0000", 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0639999999999996e-06 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030067+0000", 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1334e-05 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030095+0000", 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8300999999999999e-05 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030726+0000", 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063068100000000002 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030741+0000", 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5477000000000001e-05 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037814+0000", 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070730510000000003 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037858+0000", 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3614e-05 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4737.0:885 3.14 e37/18 3:2be3985e:::rbd_data.110c6af4f54c.0000000000000009:head v 37'38, mlcod=36'35)", 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.030122+0000", 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6742587860000002, 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0078082840000000004, 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030122+0000", 2024-05-06T00:27:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030122+0000", 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030123+0000", 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7899999999999998e-07 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030188+0000", 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5340999999999997e-05 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030200+0000", 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1396999999999999e-05 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030208+0000", 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2619999999999998e-06 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030274+0000", 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6659e-05 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030300+0000", 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5120999999999999e-05 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037869+0000", 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075697619999999998 2024-05-06T00:27:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037930+0000", 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1162999999999997e-05 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4737.0:875 3.1b e37/18 3:dd6c69eb:::rbd_data.110c6af4f54c.0000000000000004:head v 37'221, mlcod=36'218)", 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.030351+0000", 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6740303490000001, 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0077234840000000001, 2024-05-06T00:27:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030351+0000", 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030351+0000", 2024-05-06T00:27:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030351+0000", 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1200000000000003e-07 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030364+0000", 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3182e-05 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030375+0000", 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0798000000000001e-05 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030382+0000", 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0010000000000004e-06 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030585+0000", 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020310599999999999 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030608+0000", 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2327000000000002e-05 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037997+0000", 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073893240000000001 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038074+0000", 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7233999999999999e-05 2024-05-06T00:27:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4737.0:889 3.6 e37/33 3:645a233e:::rbd_data.110c6af4f54c.000000000000000b:head v 37'324, mlcod=36'321)", 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.030403+0000", 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6739775429999999, 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007649299, 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030403+0000", 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030403+0000", 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030404+0000", 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.37e-07 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030415+0000", 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1219999999999999e-05 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030425+0000", 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2220000000000008e-06 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030431+0000", 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2999999999999998e-06 2024-05-06T00:27:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030505+0000", 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4245999999999994e-05 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.030541+0000", 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5876e-05 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.037974+0000", 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074331930000000003 2024-05-06T00:27:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038053+0000", 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8504999999999995e-05 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2752 3.3 3:c4cd596e:::rbd_data.110cd55b09e5.000000000000002f:head [set-alloc-hint object_size 2097152 write_size 2097152,write 927232~18944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:13.139082+0000", 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.56529871499999995, 2024-05-06T00:27:13.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050706850000000001, 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2752 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139082+0000", 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139082+0000", 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139084+0000", 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.33e-06 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139221+0000", 2024-05-06T00:27:13.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013758600000000001 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139238+0000", 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7206000000000001e-05 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139249+0000", 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0767e-05 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139387+0000", 2024-05-06T00:27:13.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013814400000000001 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139522+0000", 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013473700000000001 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.139773+0000", 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025136100000000002 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.140981+0000", 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001207294 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.144052+0000", 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030714819999999999 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.144099+0000", 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6453999999999999e-05 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.144153+0000", 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4323999999999997e-05 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3047 3.1a 3:58574910:::rbd_data.110cd55b09e5.0000000000000010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:13.286347+0000", 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.418034032, 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005427917, 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3047 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.286347+0000", 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.286347+0000", 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.286348+0000", 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7300000000000005e-07 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.286352+0000", 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5580000000000002e-06 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.286363+0000", 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1018e-05 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.286371+0000", 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3119999999999996e-06 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.287270+0000", 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089922899999999998 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.287555+0000", 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028491999999999998 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.287852+0000", 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000297027 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.291507+0000", 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036551190000000001 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.291542+0000", 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5247999999999999e-05 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.291666+0000", 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012360100000000001 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:13.291775+0000", 2024-05-06T00:27:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010911199999999999 2024-05-06T00:27:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:13.822 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:13.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:13.824 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:13.820+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:13.893 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:27:13.894 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.0 to 1 (10000) 2024-05-06T00:27:13.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-05-06T00:27:13.925 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:13.924+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:14.007 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:14.008 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:14.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:14.064 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:14.064+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:14.127 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:14.125+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:14.189 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:14.190 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:14.190 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:14.190 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:14.190 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:14.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:14.248 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:14.248+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:14.329 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:14.329+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:14.335 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:301 3.5 3:a63c39f3:::rbd_data.110cc8beeb04.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.565971505, 2024-05-06T00:27:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010279264999999999, 2024-05-06T00:27:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 301 2024-05-06T00:27:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768675+0000", 2024-05-06T00:27:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768804+0000", 2024-05-06T00:27:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012949899999999999 2024-05-06T00:27:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768847+0000", 2024-05-06T00:27:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2953e-05 2024-05-06T00:27:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768867+0000", 2024-05-06T00:27:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9859000000000001e-05 2024-05-06T00:27:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.769041+0000", 2024-05-06T00:27:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017415900000000001 2024-05-06T00:27:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.769059+0000", 2024-05-06T00:27:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.774e-05 2024-05-06T00:27:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774608+0000", 2024-05-06T00:27:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055493640000000002 2024-05-06T00:27:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774935+0000", 2024-05-06T00:27:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000326464 2024-05-06T00:27:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.775337+0000", 2024-05-06T00:27:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040168099999999998 2024-05-06T00:27:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.777983+0000", 2024-05-06T00:27:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026467119999999998 2024-05-06T00:27:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778711+0000", 2024-05-06T00:27:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00072804800000000002 2024-05-06T00:27:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778876+0000", 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164869 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778950+0000", 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3873000000000002e-05 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:311 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.869024+0000", 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4656179820000004, 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087245489999999998, 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 311 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869024+0000", 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869024+0000", 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869026+0000", 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.877e-06 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869036+0000", 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-06 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869060+0000", 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3595e-05 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869080+0000", 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0825000000000001e-05 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870947+0000", 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001866637 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.871159+0000", 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021156899999999999 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872180+0000", 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010214550000000001 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877456+0000", 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005275503 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877495+0000", 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400999999999998e-05 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877637+0000", 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014176600000000001 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877749+0000", 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011212100000000001 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:312 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.870061+0000", 2024-05-06T00:27:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4645815400000002, 2024-05-06T00:27:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098911280000000008, 2024-05-06T00:27:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 312 2024-05-06T00:27:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870061+0000", 2024-05-06T00:27:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870061+0000", 2024-05-06T00:27:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870063+0000", 2024-05-06T00:27:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6759999999999999e-06 2024-05-06T00:27:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870078+0000", 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4079e-05 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870104+0000", 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6404999999999999e-05 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870124+0000", 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9916e-05 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873011+0000", 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028876079999999998 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873264+0000", 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000252224 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874077+0000", 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081280600000000001 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877777+0000", 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037005340000000001 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.879687+0000", 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019101610000000001 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.879841+0000", 2024-05-06T00:27:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015351199999999999 2024-05-06T00:27:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.879952+0000", 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011120699999999999 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:313 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.870424+0000", 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4642178430000001, 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.009914427, 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 313 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870424+0000", 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870424+0000", 2024-05-06T00:27:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870427+0000", 2024-05-06T00:27:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0999999999999998e-06 2024-05-06T00:27:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870438+0000", 2024-05-06T00:27:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0934999999999999e-05 2024-05-06T00:27:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870464+0000", 2024-05-06T00:27:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6169e-05 2024-05-06T00:27:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870483+0000", 2024-05-06T00:27:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8909000000000001e-05 2024-05-06T00:27:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875060+0000", 2024-05-06T00:27:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045778040000000004 2024-05-06T00:27:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875303+0000", 2024-05-06T00:27:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024273099999999999 2024-05-06T00:27:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875997+0000", 2024-05-06T00:27:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000694313 2024-05-06T00:27:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.879989+0000", 2024-05-06T00:27:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039919980000000001 2024-05-06T00:27:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880114+0000", 2024-05-06T00:27:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012451699999999999 2024-05-06T00:27:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880236+0000", 2024-05-06T00:27:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121963 2024-05-06T00:27:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880339+0000", 2024-05-06T00:27:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102988 2024-05-06T00:27:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:314 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.870515+0000", 2024-05-06T00:27:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4641270080000002, 2024-05-06T00:27:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015808361, 2024-05-06T00:27:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 314 2024-05-06T00:27:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870515+0000", 2024-05-06T00:27:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870515+0000", 2024-05-06T00:27:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870517+0000", 2024-05-06T00:27:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.373e-06 2024-05-06T00:27:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870525+0000", 2024-05-06T00:27:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1300000000000001e-06 2024-05-06T00:27:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870564+0000", 2024-05-06T00:27:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9373999999999999e-05 2024-05-06T00:27:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870580+0000", 2024-05-06T00:27:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5556000000000001e-05 2024-05-06T00:27:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877885+0000", 2024-05-06T00:27:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073051180000000002 2024-05-06T00:27:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.878108+0000", 2024-05-06T00:27:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000222878 2024-05-06T00:27:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.878797+0000", 2024-05-06T00:27:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068953299999999999 2024-05-06T00:27:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.884150+0000", 2024-05-06T00:27:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053527419999999997 2024-05-06T00:27:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886061+0000", 2024-05-06T00:27:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019113870000000001 2024-05-06T00:27:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886213+0000", 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015207600000000001 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886324+0000", 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110194 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:315 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.872437+0000", 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.462205365, 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015959788999999999, 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 315 2024-05-06T00:27:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872437+0000", 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872437+0000", 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872440+0000", 2024-05-06T00:27:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7939999999999998e-06 2024-05-06T00:27:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872454+0000", 2024-05-06T00:27:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4455e-05 2024-05-06T00:27:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872486+0000", 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1588000000000003e-05 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872509+0000", 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3129000000000002e-05 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880481+0000", 2024-05-06T00:27:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079722260000000007 2024-05-06T00:27:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880664+0000", 2024-05-06T00:27:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018270600000000001 2024-05-06T00:27:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.881567+0000", 2024-05-06T00:27:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00090353799999999995 2024-05-06T00:27:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886352+0000", 2024-05-06T00:27:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004784366 2024-05-06T00:27:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888146+0000", 2024-05-06T00:27:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001794101 2024-05-06T00:27:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888293+0000", 2024-05-06T00:27:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014686899999999999 2024-05-06T00:27:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888397+0000", 2024-05-06T00:27:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104017 2024-05-06T00:27:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:316 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.872547+0000", 2024-05-06T00:27:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.462095454, 2024-05-06T00:27:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016187126999999999, 2024-05-06T00:27:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 316 2024-05-06T00:27:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872547+0000", 2024-05-06T00:27:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872547+0000", 2024-05-06T00:27:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872549+0000", 2024-05-06T00:27:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-05-06T00:27:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872560+0000", 2024-05-06T00:27:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1096e-05 2024-05-06T00:27:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872579+0000", 2024-05-06T00:27:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9312000000000001e-05 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872594+0000", 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4486e-05 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.882373+0000", 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0097793080000000004 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.882640+0000", 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026740200000000003 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.883236+0000", 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00059588100000000004 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888435+0000", 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051989209999999996 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888551+0000", 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115667 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888653+0000", 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010173199999999999 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888734+0000", 2024-05-06T00:27:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1471999999999995e-05 2024-05-06T00:27:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:317 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.872651+0000", 2024-05-06T00:27:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4619912199999998, 2024-05-06T00:27:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016448147, 2024-05-06T00:27:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 317 2024-05-06T00:27:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872651+0000", 2024-05-06T00:27:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872651+0000", 2024-05-06T00:27:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872653+0000", 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7439999999999999e-06 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872663+0000", 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0039e-05 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872685+0000", 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2269e-05 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872701+0000", 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6016e-05 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.884310+0000", 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011608369 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.884532+0000", 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022245500000000001 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.885270+0000", 2024-05-06T00:27:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00073771000000000004 2024-05-06T00:27:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888881+0000", 2024-05-06T00:27:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003611436 2024-05-06T00:27:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888914+0000", 2024-05-06T00:27:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2741999999999999e-05 2024-05-06T00:27:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.889016+0000", 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102088 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.889099+0000", 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3279e-05 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:318 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.873382+0000", 2024-05-06T00:27:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4612608160000002, 2024-05-06T00:27:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020727737, 2024-05-06T00:27:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 318 2024-05-06T00:27:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873382+0000", 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873382+0000", 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873384+0000", 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2340000000000001e-06 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873400+0000", 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5991999999999999e-05 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873431+0000", 2024-05-06T00:27:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1421999999999998e-05 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873448+0000", 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6713000000000001e-05 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886447+0000", 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012999184 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886614+0000", 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167348 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.887292+0000", 2024-05-06T00:27:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067805999999999997 2024-05-06T00:27:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.893882+0000", 2024-05-06T00:27:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065899060000000004 2024-05-06T00:27:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.893914+0000", 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1492000000000003e-05 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.893994+0000", 2024-05-06T00:27:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0191999999999994e-05 2024-05-06T00:27:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.894109+0000", 2024-05-06T00:27:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115194 2024-05-06T00:27:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:319 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.873953+0000", 2024-05-06T00:27:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4606892729999998, 2024-05-06T00:27:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022489407999999999, 2024-05-06T00:27:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 319 2024-05-06T00:27:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873953+0000", 2024-05-06T00:27:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873953+0000", 2024-05-06T00:27:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873955+0000", 2024-05-06T00:27:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9400000000000001e-06 2024-05-06T00:27:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873964+0000", 2024-05-06T00:27:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8829999999999995e-06 2024-05-06T00:27:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873990+0000", 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6179000000000001e-05 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874008+0000", 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7691999999999999e-05 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.889327+0000", 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015319607000000001 2024-05-06T00:27:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.889541+0000", 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021325200000000001 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.890641+0000", 2024-05-06T00:27:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011003440000000001 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.894223+0000", 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003582408 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896154+0000", 2024-05-06T00:27:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001930736 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896320+0000", 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016567099999999999 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896442+0000", 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012269599999999999 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:320 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.874567+0000", 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4600755660000004, 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022359845, 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 320 2024-05-06T00:27:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874567+0000", 2024-05-06T00:27:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874567+0000", 2024-05-06T00:27:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874570+0000", 2024-05-06T00:27:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9129999999999999e-06 2024-05-06T00:27:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874584+0000", 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4457000000000001e-05 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874619+0000", 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4448e-05 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874640+0000", 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1118999999999999e-05 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.891755+0000", 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017115086000000002 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.891954+0000", 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019934 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.892960+0000", 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001006369 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896483+0000", 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035224010000000001 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896604+0000", 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121006 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896824+0000", 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022020199999999999 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896927+0000", 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010250400000000001 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:321 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.875133+0000", 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4595094959999999, 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023228217999999998, 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 321 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875133+0000", 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875133+0000", 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875135+0000", 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.86e-06 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875144+0000", 2024-05-06T00:27:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6490000000000003e-06 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875165+0000", 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0503e-05 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875177+0000", 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2646000000000001e-05 2024-05-06T00:27:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.894287+0000", 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019109206 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.894427+0000", 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014071 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.895143+0000", 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00071604100000000005 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896969+0000", 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001825478 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.898129+0000", 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001160183 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.898269+0000", 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013992099999999999 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.898361+0000", 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2021000000000005e-05 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:559 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.082786+0000", 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2518558349999998, 2024-05-06T00:27:14.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085290509999999993, 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 559 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082786+0000", 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082786+0000", 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082788+0000", 2024-05-06T00:27:14.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8339999999999999e-06 2024-05-06T00:27:14.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082797+0000", 2024-05-06T00:27:14.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9789999999999999e-06 2024-05-06T00:27:14.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082818+0000", 2024-05-06T00:27:14.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0449e-05 2024-05-06T00:27:14.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082840+0000", 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2325999999999999e-05 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084697+0000", 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018566310000000001 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084941+0000", 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000244164 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085882+0000", 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00094077400000000002 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.089326+0000", 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034444150000000002 2024-05-06T00:27:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091071+0000", 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001745251 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091209+0000", 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000137403 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091316+0000", 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010682499999999999 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:560 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.082873+0000", 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2517696850000002, 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011230254, 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.475 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 560 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082873+0000", 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082873+0000", 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082874+0000", 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.519e-06 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082881+0000", 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2359999999999998e-06 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082905+0000", 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3723e-05 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082923+0000", 2024-05-06T00:27:14.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7685000000000001e-05 2024-05-06T00:27:14.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.086807+0000", 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038844249999999999 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.087035+0000", 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022779199999999999 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.087751+0000", 2024-05-06T00:27:14.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00071593500000000001 2024-05-06T00:27:14.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.093832+0000", 2024-05-06T00:27:14.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00608137 2024-05-06T00:27:14.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.093872+0000", 2024-05-06T00:27:14.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9755000000000001e-05 2024-05-06T00:27:14.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.093998+0000", 2024-05-06T00:27:14.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012544300000000001 2024-05-06T00:27:14.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.094103+0000", 2024-05-06T00:27:14.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105371 2024-05-06T00:27:14.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.484 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:561 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.083118+0000", 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2515240319999998, 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013481519000000001, 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.486 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 561 2024-05-06T00:27:14.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.486 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083118+0000", 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083118+0000", 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083120+0000", 2024-05-06T00:27:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8330000000000001e-06 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083130+0000", 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6860000000000001e-06 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083157+0000", 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6789000000000001e-05 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083177+0000", 2024-05-06T00:27:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0781999999999999e-05 2024-05-06T00:27:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.089474+0000", 2024-05-06T00:27:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062963079999999996 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.089656+0000", 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000182561 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.090298+0000", 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00064222900000000002 2024-05-06T00:27:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.094401+0000", 2024-05-06T00:27:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041023639999999998 2024-05-06T00:27:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096354+0000", 2024-05-06T00:27:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001953097 2024-05-06T00:27:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096484+0000", 2024-05-06T00:27:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013024600000000001 2024-05-06T00:27:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096600+0000", 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115624 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:570 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.083568+0000", 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2510742119999998, 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015897214999999999, 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 570 2024-05-06T00:27:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083568+0000", 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083568+0000", 2024-05-06T00:27:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083570+0000", 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083581+0000", 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0392e-05 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083608+0000", 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6783000000000001e-05 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083627+0000", 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9779000000000002e-05 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091442+0000", 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0078141879999999997 2024-05-06T00:27:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091738+0000", 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029659400000000001 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.092803+0000", 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010646270000000001 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096632+0000", 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038287680000000002 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099233+0000", 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026018700000000001 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099370+0000", 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136339 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099465+0000", 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5521999999999998e-05 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:571 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.084855+0000", 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2497875670000003, 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014876495999999999, 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 571 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084855+0000", 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084855+0000", 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084857+0000", 2024-05-06T00:27:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4329999999999998e-06 2024-05-06T00:27:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084869+0000", 2024-05-06T00:27:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1966e-05 2024-05-06T00:27:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084896+0000", 2024-05-06T00:27:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6441999999999999e-05 2024-05-06T00:27:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084910+0000", 2024-05-06T00:27:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4543e-05 2024-05-06T00:27:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.094566+0000", 2024-05-06T00:27:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096554580000000004 2024-05-06T00:27:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.094732+0000", 2024-05-06T00:27:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016648100000000001 2024-05-06T00:27:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.095350+0000", 2024-05-06T00:27:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061754699999999997 2024-05-06T00:27:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099539+0000", 2024-05-06T00:27:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041893290000000003 2024-05-06T00:27:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099574+0000", 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4647999999999998e-05 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099647+0000", 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3570000000000002e-05 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099731+0000", 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4079000000000006e-05 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:580 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.085074+0000", 2024-05-06T00:27:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2495679060000002, 2024-05-06T00:27:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015523696, 2024-05-06T00:27:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 580 2024-05-06T00:27:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085074+0000", 2024-05-06T00:27:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085074+0000", 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085076+0000", 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6199999999999999e-06 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085084+0000", 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4540000000000007e-06 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085101+0000", 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6545000000000002e-05 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085111+0000", 2024-05-06T00:27:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0189e-05 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096709+0000", 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011597573 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096885+0000", 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017651099999999999 2024-05-06T00:27:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.098140+0000", 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001254783 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.100126+0000", 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019856320000000002 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.100454+0000", 2024-05-06T00:27:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032824299999999998 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.100539+0000", 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4958999999999998e-05 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.100598+0000", 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9187000000000001e-05 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:870 3.10 3:09c2179a:::rbd_data.110c6af4f54c.0000000000000001:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.028008+0000", 2024-05-06T00:27:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.3066347340000002, 2024-05-06T00:27:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089563750000000008, 2024-05-06T00:27:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 870 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028008+0000", 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028008+0000", 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028009+0000", 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0920000000000001e-06 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028014+0000", 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6760000000000004e-06 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028030+0000", 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5770999999999999e-05 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028038+0000", 2024-05-06T00:27:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1880000000000002e-06 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028100+0000", 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1240000000000003e-05 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028169+0000", 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9183999999999994e-05 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028348+0000", 2024-05-06T00:27:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017878499999999999 2024-05-06T00:27:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.033228+0000", 2024-05-06T00:27:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004880087 2024-05-06T00:27:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.036862+0000", 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036341099999999999 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.036918+0000", 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6029000000000002e-05 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.036964+0000", 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6213e-05 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:873 3.16 3:6d311e83:::rbd_data.110c6af4f54c.0000000000000003:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.3064604529999997, 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010204229, 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 873 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028183+0000", 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4199999999999995e-07 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028185+0000", 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9720000000000001e-06 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028192+0000", 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5119999999999997e-06 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028197+0000", 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6369999999999998e-06 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028254+0000", 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7014e-05 2024-05-06T00:27:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028337+0000", 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3403000000000001e-05 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028632+0000", 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029496999999999998 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.032702+0000", 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040701230000000001 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038251+0000", 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005549099 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038332+0000", 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0694999999999999e-05 2024-05-06T00:27:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038386+0000", 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4162000000000001e-05 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:14.534 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:14.535 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-05-06T00:27:14.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:14.540 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:14.537+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:14.549 INFO:teuthology.orchestra.run.smithi186.stdout:1316 write 0x5e80600 thru 0x5e8fd45 (0xf746 bytes) 2024-05-06T00:27:14.552 INFO:teuthology.orchestra.run.smithi186.stdout:1317 read 0x4d6dc00 thru 0x4d7614c (0x854d bytes) 2024-05-06T00:27:14.552 INFO:teuthology.orchestra.run.smithi186.stdout:1318 write 0x1c49a00 thru 0x1c55910 (0xbf11 bytes) 2024-05-06T00:27:14.553 INFO:teuthology.orchestra.run.smithi186.stdout:1319 read 0xc14000 thru 0xc1aa01 (0x6a02 bytes) 2024-05-06T00:27:14.553 INFO:teuthology.orchestra.run.smithi186.stdout:1321 punch from 0x1097600 to 0x10a2400, (0xae00 bytes) 2024-05-06T00:27:14.578 INFO:teuthology.orchestra.run.smithi186.stdout:1324 trunc from 0x5e8fe00 to 0x949cc00 2024-05-06T00:27:14.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:14.623 INFO:teuthology.orchestra.run.smithi186.stdout:1327 trunc from 0x949cc00 to 0xaf200 2024-05-06T00:27:14.650 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:14.649+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:14.686 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:14.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:14.686 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:14.686 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:14.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:14.701 INFO:teuthology.orchestra.run.smithi186.stdout:1328 trunc from 0xaf200 to 0xad3aa00 2024-05-06T00:27:14.716 INFO:teuthology.orchestra.run.smithi186.stdout:1329 read 0xa77a400 thru 0xa7805e8 (0x61e9 bytes) 2024-05-06T00:27:14.719 INFO:teuthology.orchestra.run.smithi186.stdout:1330 write 0x46bbe00 thru 0x46c595b (0x9b5c bytes) 2024-05-06T00:27:14.719 INFO:teuthology.orchestra.run.smithi186.stdout:1335 write 0x14a1e00 thru 0x14a58c1 (0x3ac2 bytes) 2024-05-06T00:27:14.719 INFO:teuthology.orchestra.run.smithi186.stdout:1336 read 0x38adc00 thru 0x38b8eb8 (0xb2b9 bytes) 2024-05-06T00:27:14.720 INFO:teuthology.orchestra.run.smithi186.stdout:1339 read 0x15a9600 thru 0x15b4544 (0xaf45 bytes) 2024-05-06T00:27:14.720 INFO:teuthology.orchestra.run.smithi186.stdout:1342 read 0x11f5600 thru 0x11fb446 (0x5e47 bytes) 2024-05-06T00:27:14.729 INFO:teuthology.orchestra.run.smithi186.stdout:1343 trunc from 0xad3aa00 to 0x650a200 2024-05-06T00:27:14.739 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:14.736+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:14.765 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:14.765 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:14.779 INFO:teuthology.orchestra.run.smithi186.stdout:1344 write 0x86e0c00 thru 0x86f0b36 (0xff37 bytes) 2024-05-06T00:27:14.781 INFO:teuthology.orchestra.run.smithi186.stdout:1346 read 0x2bec00 thru 0x2c2254 (0x3655 bytes) 2024-05-06T00:27:14.782 INFO:teuthology.orchestra.run.smithi186.stdout:1347 punch from 0x38b6000 to 0x38c0400, (0xa400 bytes) 2024-05-06T00:27:14.782 INFO:teuthology.orchestra.run.smithi186.stdout:1348 write 0x2c9be00 thru 0x2ca09b9 (0x4bba bytes) 2024-05-06T00:27:14.782 INFO:teuthology.orchestra.run.smithi186.stdout:1353 punch from 0x313cc00 to 0x313de00, (0x1200 bytes) 2024-05-06T00:27:14.782 INFO:teuthology.orchestra.run.smithi186.stdout:1356 write 0x23a5000 thru 0x23abef4 (0x6ef5 bytes) 2024-05-06T00:27:14.790 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:14.807 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:14.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:14.827 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:14.838 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:14.837+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:14.850 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:14.851 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:14.852 INFO:teuthology.orchestra.run.smithi186.stdout:1357 write 0xe716600 thru 0xe71e28e (0x7c8f bytes) 2024-05-06T00:27:14.856 INFO:teuthology.orchestra.run.smithi186.stdout:1360 read 0x4a9d200 thru 0x4aa70a1 (0x9ea2 bytes) 2024-05-06T00:27:14.867 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:14.867 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:14.870 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:14.871 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:14.873 INFO:teuthology.orchestra.run.smithi186.stdout:1363 trunc from 0xe71e400 to 0x73e9400 2024-05-06T00:27:14.884 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.1 to 1 (10000) 2024-05-06T00:27:14.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-05-06T00:27:14.966 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:14.965+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:15.016 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:15.016 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:15.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:15.039 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:15.040+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:15.141 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:15.137+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:15.150 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:15.150 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:15.157 INFO:teuthology.orchestra.run.smithi186.stdout:1365 write 0x4fa5a00 thru 0x4fae82a (0x8e2b bytes) 2024-05-06T00:27:15.157 INFO:teuthology.orchestra.run.smithi186.stdout:1366 read 0x3c52e00 thru 0x3c61a9a (0xec9b bytes) 2024-05-06T00:27:15.158 INFO:teuthology.orchestra.run.smithi186.stdout:1367 read 0x4f6d800 thru 0x4f75eb4 (0x86b5 bytes) 2024-05-06T00:27:15.158 INFO:teuthology.orchestra.run.smithi186.stdout:1368 read 0x2dad000 thru 0x2dbab99 (0xdb9a bytes) 2024-05-06T00:27:15.165 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:15.191 INFO:teuthology.orchestra.run.smithi186.stdout:1371 write 0xc44ca00 thru 0xc450e80 (0x4481 bytes) 2024-05-06T00:27:15.195 INFO:teuthology.orchestra.run.smithi186.stdout:1373 trunc from 0xc451000 to 0xb254c00 2024-05-06T00:27:15.241 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:15.240+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:15.250 INFO:teuthology.orchestra.run.smithi186.stdout:1377 trunc from 0xb254c00 to 0x2285000 2024-05-06T00:27:15.300 INFO:teuthology.orchestra.run.smithi186.stdout:1378 write 0x4b83200 thru 0x4b898d5 (0x66d6 bytes) 2024-05-06T00:27:15.304 INFO:teuthology.orchestra.run.smithi186.stdout:1379 trunc from 0x4b89a00 to 0x5e9ac00 2024-05-06T00:27:15.334 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:15.334 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:15.342 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:15.341+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:15.344 INFO:teuthology.orchestra.run.smithi186.stdout:1380 write 0x837c200 thru 0x8384604 (0x8405 bytes) 2024-05-06T00:27:15.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:15.353 INFO:teuthology.orchestra.run.smithi186.stdout:1381 trunc from 0x8384800 to 0x9c3b800 2024-05-06T00:27:15.373 INFO:teuthology.orchestra.run.smithi186.stdout:1383 punch from 0x6d5b200 to 0x6d5ce00, (0x1c00 bytes) 2024-05-06T00:27:15.375 INFO:teuthology.orchestra.run.smithi186.stdout:1384 write 0x7b9e200 thru 0x7ba812a (0x9f2b bytes) 2024-05-06T00:27:15.376 INFO:teuthology.orchestra.run.smithi186.stdout:1385 punch from 0x8680800 to 0x8684000, (0x3800 bytes) 2024-05-06T00:27:15.396 INFO:teuthology.orchestra.run.smithi186.stdout:1386 trunc from 0x9c3b800 to 0x1274a00 2024-05-06T00:27:15.432 INFO:teuthology.orchestra.run.smithi186.stdout:1387 trunc from 0x1274a00 to 0x7b2c600 2024-05-06T00:27:15.443 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:15.440+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:15.451 INFO:teuthology.orchestra.run.smithi186.stdout:1388 read 0x1214800 thru 0x121f259 (0xaa5a bytes) 2024-05-06T00:27:15.452 INFO:teuthology.orchestra.run.smithi186.stdout:1389 read 0x36a9800 thru 0x36aaf1d (0x171e bytes) 2024-05-06T00:27:15.453 INFO:teuthology.orchestra.run.smithi186.stdout:1392 punch from 0x57e0800 to 0x57e3600, (0x2e00 bytes) 2024-05-06T00:27:15.453 INFO:teuthology.orchestra.run.smithi186.stdout:1393 read 0x3e1e400 thru 0x3e20c9c (0x289d bytes) 2024-05-06T00:27:15.453 INFO:teuthology.orchestra.run.smithi186.stdout:1396 read 0x2c42200 thru 0x2c432f2 (0x10f3 bytes) 2024-05-06T00:27:15.453 INFO:teuthology.orchestra.run.smithi186.stdout:1397 read 0x55f2000 thru 0x55f54de (0x34df bytes) 2024-05-06T00:27:15.454 INFO:teuthology.orchestra.run.smithi186.stdout:1398 read 0x928200 thru 0x92b746 (0x3547 bytes) 2024-05-06T00:27:15.456 INFO:teuthology.orchestra.run.smithi186.stdout:1399 trunc from 0x7b2c600 to 0x6ad9600 2024-05-06T00:27:15.476 INFO:teuthology.orchestra.run.smithi186.stdout:1401 read 0x1e21a00 thru 0x1e3097b (0xef7c bytes) 2024-05-06T00:27:15.481 INFO:teuthology.orchestra.run.smithi186.stdout:1405 write 0x436b600 thru 0x43753b0 (0x9db1 bytes) 2024-05-06T00:27:15.481 INFO:teuthology.orchestra.run.smithi186.stdout:1406 write 0x3f5200 thru 0x3f9d56 (0x4b57 bytes) 2024-05-06T00:27:15.481 INFO:teuthology.orchestra.run.smithi186.stdout:1407 punch from 0x4a45200 to 0x4a4e800, (0x9600 bytes) 2024-05-06T00:27:15.512 INFO:teuthology.orchestra.run.smithi186.stdout:1409 write 0x9d32c00 thru 0x9d3ec0d (0xc00e bytes) 2024-05-06T00:27:15.513 INFO:teuthology.orchestra.run.smithi186.stdout:1412 write 0x3b79c00 thru 0x3b7ee40 (0x5241 bytes) 2024-05-06T00:27:15.513 INFO:teuthology.orchestra.run.smithi186.stdout:1413 read 0x5995c00 thru 0x59a4934 (0xed35 bytes) 2024-05-06T00:27:15.513 INFO:teuthology.orchestra.run.smithi186.stdout:1414 punch from 0x45a3c00 to 0x45ad200, (0x9600 bytes) 2024-05-06T00:27:15.514 INFO:teuthology.orchestra.run.smithi186.stdout:1415 punch from 0x61fa800 to 0x61ffa00, (0x5200 bytes) 2024-05-06T00:27:15.515 INFO:teuthology.orchestra.run.smithi186.stdout:1416 read 0x1035a00 thru 0x1038c28 (0x3229 bytes) 2024-05-06T00:27:15.515 INFO:teuthology.orchestra.run.smithi186.stdout:1419 write 0x1c92c00 thru 0x1c9635a (0x375b bytes) 2024-05-06T00:27:15.515 INFO:teuthology.orchestra.run.smithi186.stdout:1422 punch from 0x9879e00 to 0x987ba00, (0x1c00 bytes) 2024-05-06T00:27:15.515 INFO:teuthology.orchestra.run.smithi186.stdout:1424 read 0x50bfc00 thru 0x50ce74f (0xeb50 bytes) 2024-05-06T00:27:15.516 INFO:teuthology.orchestra.run.smithi186.stdout:1425 write 0x2a22600 thru 0x2a28b27 (0x6528 bytes) 2024-05-06T00:27:15.519 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:15.519 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:15.524 INFO:teuthology.orchestra.run.smithi186.stdout:1426 trunc from 0x9d3ee00 to 0x5baaa00 2024-05-06T00:27:15.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:15.545 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:15.544+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:15.551 INFO:teuthology.orchestra.run.smithi186.stdout:1427 punch from 0x3da7c00 to 0x3da8c00, (0x1000 bytes) 2024-05-06T00:27:15.553 INFO:teuthology.orchestra.run.smithi186.stdout:1428 read 0x2046c00 thru 0x20501ef (0x95f0 bytes) 2024-05-06T00:27:15.560 INFO:teuthology.orchestra.run.smithi186.stdout:1429 trunc from 0x5baaa00 to 0x2347400 2024-05-06T00:27:15.619 INFO:teuthology.orchestra.run.smithi186.stdout:1430 write 0x8fe5400 thru 0x8fea223 (0x4e24 bytes) 2024-05-06T00:27:15.620 INFO:teuthology.orchestra.run.smithi186.stdout:1432 write 0xc32e00 thru 0xc3725b (0x445c bytes) 2024-05-06T00:27:15.620 INFO:teuthology.orchestra.run.smithi186.stdout:1434 write 0x3901000 thru 0x3907499 (0x649a bytes) 2024-05-06T00:27:15.621 INFO:teuthology.orchestra.run.smithi186.stdout:1435 write 0x7093000 thru 0x709c153 (0x9154 bytes) 2024-05-06T00:27:15.646 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:15.644+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:15.657 INFO:teuthology.orchestra.run.smithi186.stdout:1436 write 0xbb50400 thru 0xbb51c39 (0x183a bytes) 2024-05-06T00:27:15.658 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:15.658 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:15.671 INFO:teuthology.orchestra.run.smithi186.stdout:1437 trunc from 0xbb51e00 to 0x647b800 2024-05-06T00:27:15.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:15.701 INFO:teuthology.orchestra.run.smithi186.stdout:1438 write 0x5fa8800 thru 0x5faeb4f (0x6350 bytes) 2024-05-06T00:27:15.742 INFO:teuthology.orchestra.run.smithi186.stdout:1440 write 0xe5d4e00 thru 0xe5db1ee (0x63ef bytes) 2024-05-06T00:27:15.746 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:15.744+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:15.754 INFO:teuthology.orchestra.run.smithi186.stdout:1441 trunc from 0xe5db200 to 0xa342c00 2024-05-06T00:27:15.794 INFO:teuthology.orchestra.run.smithi186.stdout:1443 punch from 0x9d59800 to 0x9d5da00, (0x4200 bytes) 2024-05-06T00:27:15.800 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:15.800 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:15.804 INFO:teuthology.orchestra.run.smithi186.stdout:1444 trunc from 0xa342c00 to 0x6cbae00 2024-05-06T00:27:15.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:15.849 INFO:teuthology.orchestra.run.smithi186.stdout:1445 trunc from 0x6cbae00 to 0xd452c00 2024-05-06T00:27:15.861 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:15.861+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:15.864 INFO:teuthology.orchestra.run.smithi186.stdout:1446 punch from 0x1ebe800 to 0x1ec2c00, (0x4400 bytes) 2024-05-06T00:27:15.867 INFO:teuthology.orchestra.run.smithi186.stdout:1450 write 0x7d3800 thru 0x7e02c8 (0xcac9 bytes) 2024-05-06T00:27:15.888 INFO:teuthology.orchestra.run.smithi186.stdout:1452 write 0xd500e00 thru 0xd5105d8 (0xf7d9 bytes) 2024-05-06T00:27:15.889 INFO:teuthology.orchestra.run.smithi186.stdout:1455 punch from 0xc88ca00 to 0xc897400, (0xaa00 bytes) 2024-05-06T00:27:15.890 INFO:teuthology.orchestra.run.smithi186.stdout:1457 read 0x8160e00 thru 0x81701ef (0xf3f0 bytes) 2024-05-06T00:27:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.2 to 1 (10000) 2024-05-06T00:27:15.909 INFO:teuthology.orchestra.run.smithi186.stdout:1459 trunc from 0xd510600 to 0x2685400 2024-05-06T00:27:15.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:27:15.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-05-06T00:27:15.943 INFO:teuthology.orchestra.run.smithi186.stdout:1461 trunc from 0x2685400 to 0x13bc00 2024-05-06T00:27:15.966 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:15.966 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:15.966 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:15.965+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:15.984 INFO:teuthology.orchestra.run.smithi186.stdout:1463 trunc from 0x13bc00 to 0x302fc00 2024-05-06T00:27:15.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:16.014 INFO:teuthology.orchestra.run.smithi186.stdout:1465 write 0x5d75600 thru 0x5d81824 (0xc225 bytes) 2024-05-06T00:27:16.016 INFO:teuthology.orchestra.run.smithi186.stdout:1468 read 0x3de6a00 thru 0x3deef6d (0x856e bytes) 2024-05-06T00:27:16.016 INFO:teuthology.orchestra.run.smithi186.stdout:1469 punch from 0x2fb6400 to 0x2fb9a00, (0x3600 bytes) 2024-05-06T00:27:16.020 INFO:teuthology.orchestra.run.smithi186.stdout:1470 trunc from 0x5d81a00 to 0x4862a00 2024-05-06T00:27:16.039 INFO:teuthology.orchestra.run.smithi186.stdout:1472 write 0x4295800 thru 0x429ce3b (0x763c bytes) 2024-05-06T00:27:16.040 INFO:teuthology.orchestra.run.smithi186.stdout:1474 read 0x203b600 thru 0x203c939 (0x133a bytes) 2024-05-06T00:27:16.049 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:16.048+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:16.066 INFO:teuthology.orchestra.run.smithi186.stdout:1477 write 0x969fa00 thru 0x96abdae (0xc3af bytes) 2024-05-06T00:27:16.076 INFO:teuthology.orchestra.run.smithi186.stdout:1478 trunc from 0x96abe00 to 0x5a45600 2024-05-06T00:27:16.123 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:16.123 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:16.135 INFO:teuthology.orchestra.run.smithi186.stdout:1479 write 0xcacda00 thru 0xcad44f9 (0x6afa bytes) 2024-05-06T00:27:16.137 INFO:teuthology.orchestra.run.smithi186.stdout:1481 write 0x6826200 thru 0x6833350 (0xd151 bytes) 2024-05-06T00:27:16.137 INFO:teuthology.orchestra.run.smithi186.stdout:1484 read 0x3422400 thru 0x342f8fb (0xd4fc bytes) 2024-05-06T00:27:16.138 INFO:teuthology.orchestra.run.smithi186.stdout:1486 trunc from 0xcad4600 to 0xcd25000 2024-05-06T00:27:16.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:16.149 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:16.149+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:16.158 INFO:teuthology.orchestra.run.smithi186.stdout:1487 read 0x4c6fe00 thru 0x4c739f1 (0x3bf2 bytes) 2024-05-06T00:27:16.159 INFO:teuthology.orchestra.run.smithi186.stdout:1488 write 0xaccba00 thru 0xacdaeae (0xf4af bytes) 2024-05-06T00:27:16.160 INFO:teuthology.orchestra.run.smithi186.stdout:1489 write 0x3195000 thru 0x3195823 (0x824 bytes) 2024-05-06T00:27:16.160 INFO:teuthology.orchestra.run.smithi186.stdout:1490 read 0x45b1a00 thru 0x45bc488 (0xaa89 bytes) 2024-05-06T00:27:16.160 INFO:teuthology.orchestra.run.smithi186.stdout:1491 read 0x9563200 thru 0x9565f56 (0x2d57 bytes) 2024-05-06T00:27:16.160 INFO:teuthology.orchestra.run.smithi186.stdout:1493 write 0x5e24800 thru 0x5e278a7 (0x30a8 bytes) 2024-05-06T00:27:16.160 INFO:teuthology.orchestra.run.smithi186.stdout:1495 punch from 0x29d5c00 to 0x29e4000, (0xe400 bytes) 2024-05-06T00:27:16.161 INFO:teuthology.orchestra.run.smithi186.stdout:1496 write 0xa85bc00 thru 0xa867642 (0xba43 bytes) 2024-05-06T00:27:16.166 INFO:teuthology.orchestra.run.smithi186.stdout:1497 trunc from 0xcd25000 to 0xabae400 2024-05-06T00:27:16.192 INFO:teuthology.orchestra.run.smithi186.stdout:1498 write 0x2650000 thru 0x26593d0 (0x93d1 bytes) 2024-05-06T00:27:16.194 INFO:teuthology.orchestra.run.smithi186.stdout:1499 write 0x69f9600 thru 0x69fd806 (0x4207 bytes) 2024-05-06T00:27:16.194 INFO:teuthology.orchestra.run.smithi186.stdout:1502 read 0x49e7400 thru 0x49f4aef (0xd6f0 bytes) 2024-05-06T00:27:16.215 INFO:teuthology.orchestra.run.smithi186.stdout:1508 write 0xae61800 thru 0xae66155 (0x4956 bytes) 2024-05-06T00:27:16.216 INFO:teuthology.orchestra.run.smithi186.stdout:1510 read 0x4253600 thru 0x4261a91 (0xe492 bytes) 2024-05-06T00:27:16.225 INFO:teuthology.orchestra.run.smithi186.stdout:1511 trunc from 0xae66200 to 0x6568400 2024-05-06T00:27:16.250 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:16.248+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:16.260 INFO:teuthology.orchestra.run.smithi186.stdout:1513 read 0x46a1200 thru 0x46afd29 (0xeb2a bytes) 2024-05-06T00:27:16.263 INFO:teuthology.orchestra.run.smithi186.stdout:1519 punch from 0x57b2400 to 0x57bc600, (0xa200 bytes) 2024-05-06T00:27:16.263 INFO:teuthology.orchestra.run.smithi186.stdout:1520 clone 10 order 20 su 4096 sc 8 2024-05-06T00:27:16.310 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:16.310 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:16.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:16.352 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:16.349+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:16.453 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:16.453+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:16.488 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:16.488 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:16.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:16.554 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:16.553+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:16.618 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:16.618 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:16.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:16.676 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:16.676+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:16.748 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:16.748 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:16.756 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:16.756+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:16.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:16.857 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:16.856+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:16.894 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.3 to 1 (10000) 2024-05-06T00:27:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:27:16.930 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:16.931 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:16.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:16.949 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-05-06T00:27:16.988 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:16.988+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:17.041 INFO:teuthology.orchestra.run.smithi186.stdout:flattening image image_client.0-clone9 2024-05-06T00:27:17.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:17.055 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:17.058 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:17.057+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:17.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:17.072 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:17.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:17.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:17.108 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:17.115 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:17.115 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:17.129 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:17.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:17.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:17.149 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:17.159 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:17.157+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:17.257 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:17.257 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:17.257 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:17.257 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:17.257 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:17.260 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:17.260+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:17.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:17.361 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:17.360+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:17.450 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:17.450 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:17.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:17.450 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:17.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.450 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4578.0:301 3.5 3:a63c39f3:::rbd_data.110cc8beeb04.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3411968~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.678659802999999, 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010279264999999999, 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4578", 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 301 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768671+0000", 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768675+0000", 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768804+0000", 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012949899999999999 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768847+0000", 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2953e-05 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.768867+0000", 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9859000000000001e-05 2024-05-06T00:27:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.769041+0000", 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017415900000000001 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.769059+0000", 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.774e-05 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774608+0000", 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055493640000000002 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.774935+0000", 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000326464 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.775337+0000", 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040168099999999998 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.777983+0000", 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026467119999999998 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778711+0000", 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00072804800000000002 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778876+0000", 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164869 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:55.778950+0000", 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3873000000000002e-05 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:311 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.869024+0000", 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5783062799999996, 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087245489999999998, 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.457 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 311 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869024+0000", 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869024+0000", 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869026+0000", 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.877e-06 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869036+0000", 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-06 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869060+0000", 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3595e-05 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.869080+0000", 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0825000000000001e-05 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870947+0000", 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001866637 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.871159+0000", 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021156899999999999 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872180+0000", 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010214550000000001 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877456+0000", 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005275503 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877495+0000", 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400999999999998e-05 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877637+0000", 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014176600000000001 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877749+0000", 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011212100000000001 2024-05-06T00:27:17.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:312 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.870061+0000", 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5772698379999994, 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098911280000000008, 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 312 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870061+0000", 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870061+0000", 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870063+0000", 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6759999999999999e-06 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870078+0000", 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4079e-05 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870104+0000", 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6404999999999999e-05 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870124+0000", 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9916e-05 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873011+0000", 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028876079999999998 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873264+0000", 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000252224 2024-05-06T00:27:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874077+0000", 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081280600000000001 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877777+0000", 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037005340000000001 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.879687+0000", 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019101610000000001 2024-05-06T00:27:17.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.879841+0000", 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015351199999999999 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.879952+0000", 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011120699999999999 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:313 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.870424+0000", 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5769061410000003, 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.009914427, 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 313 2024-05-06T00:27:17.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870424+0000", 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870424+0000", 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870427+0000", 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0999999999999998e-06 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870438+0000", 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0934999999999999e-05 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870464+0000", 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6169e-05 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870483+0000", 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8909000000000001e-05 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875060+0000", 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045778040000000004 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875303+0000", 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024273099999999999 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875997+0000", 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000694313 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.879989+0000", 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039919980000000001 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880114+0000", 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012451699999999999 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880236+0000", 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121963 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880339+0000", 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102988 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:314 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.870515+0000", 2024-05-06T00:27:17.474 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5768153060000003, 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015808361, 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 314 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870515+0000", 2024-05-06T00:27:17.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870515+0000", 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870517+0000", 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.373e-06 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870525+0000", 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1300000000000001e-06 2024-05-06T00:27:17.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870564+0000", 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9373999999999999e-05 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.870580+0000", 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5556000000000001e-05 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.877885+0000", 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073051180000000002 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.878108+0000", 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000222878 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.878797+0000", 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068953299999999999 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.884150+0000", 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053527419999999997 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886061+0000", 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019113870000000001 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886213+0000", 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015207600000000001 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886324+0000", 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110194 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:315 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.872437+0000", 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5748936629999992, 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015959788999999999, 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 315 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872437+0000", 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872437+0000", 2024-05-06T00:27:17.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872440+0000", 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7939999999999998e-06 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872454+0000", 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4455e-05 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872486+0000", 2024-05-06T00:27:17.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1588000000000003e-05 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872509+0000", 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3129000000000002e-05 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880481+0000", 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079722260000000007 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.880664+0000", 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018270600000000001 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.881567+0000", 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00090353799999999995 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886352+0000", 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004784366 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888146+0000", 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001794101 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888293+0000", 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014686899999999999 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888397+0000", 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104017 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:316 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.872547+0000", 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5747837520000001, 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016187126999999999, 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 316 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.486 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872547+0000", 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872547+0000", 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872549+0000", 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-05-06T00:27:17.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872560+0000", 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1096e-05 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872579+0000", 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9312000000000001e-05 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872594+0000", 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4486e-05 2024-05-06T00:27:17.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.882373+0000", 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0097793080000000004 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.882640+0000", 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026740200000000003 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.883236+0000", 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00059588100000000004 2024-05-06T00:27:17.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888435+0000", 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051989209999999996 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888551+0000", 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115667 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888653+0000", 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010173199999999999 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888734+0000", 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1471999999999995e-05 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:317 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.872651+0000", 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5746795179999999, 2024-05-06T00:27:17.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016448147, 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 317 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872651+0000", 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872651+0000", 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872653+0000", 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7439999999999999e-06 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872663+0000", 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0039e-05 2024-05-06T00:27:17.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872685+0000", 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2269e-05 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.872701+0000", 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6016e-05 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.884310+0000", 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011608369 2024-05-06T00:27:17.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.884532+0000", 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022245500000000001 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.885270+0000", 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00073771000000000004 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888881+0000", 2024-05-06T00:27:17.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003611436 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.888914+0000", 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2741999999999999e-05 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.889016+0000", 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102088 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.889099+0000", 2024-05-06T00:27:17.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3279e-05 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:318 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.873382+0000", 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5739491139999995, 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020727737, 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 318 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873382+0000", 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873382+0000", 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873384+0000", 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2340000000000001e-06 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873400+0000", 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5991999999999999e-05 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873431+0000", 2024-05-06T00:27:17.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1421999999999998e-05 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873448+0000", 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6713000000000001e-05 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886447+0000", 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012999184 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.886614+0000", 2024-05-06T00:27:17.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167348 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.887292+0000", 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067805999999999997 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.893882+0000", 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065899060000000004 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.893914+0000", 2024-05-06T00:27:17.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1492000000000003e-05 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.893994+0000", 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0191999999999994e-05 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.894109+0000", 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115194 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:319 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.873953+0000", 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.573377571, 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022489407999999999, 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 319 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873953+0000", 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873953+0000", 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873955+0000", 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9400000000000001e-06 2024-05-06T00:27:17.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873964+0000", 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8829999999999995e-06 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.873990+0000", 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6179000000000001e-05 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874008+0000", 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7691999999999999e-05 2024-05-06T00:27:17.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.889327+0000", 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015319607000000001 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.889541+0000", 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021325200000000001 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.890641+0000", 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011003440000000001 2024-05-06T00:27:17.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.894223+0000", 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003582408 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896154+0000", 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001930736 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896320+0000", 2024-05-06T00:27:17.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016567099999999999 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896442+0000", 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012269599999999999 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:320 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.874567+0000", 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5727638640000006, 2024-05-06T00:27:17.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022359845, 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 320 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874567+0000", 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874567+0000", 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874570+0000", 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9129999999999999e-06 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874584+0000", 2024-05-06T00:27:17.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4457000000000001e-05 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874619+0000", 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4448e-05 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.874640+0000", 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1118999999999999e-05 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.891755+0000", 2024-05-06T00:27:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017115086000000002 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.891954+0000", 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019934 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.892960+0000", 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001006369 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896483+0000", 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035224010000000001 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896604+0000", 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121006 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896824+0000", 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022020199999999999 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896927+0000", 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010250400000000001 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:321 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:08.875133+0000", 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5721977940000009, 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023228217999999998, 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 321 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875133+0000", 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875133+0000", 2024-05-06T00:27:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875135+0000", 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.86e-06 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875144+0000", 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6490000000000003e-06 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875165+0000", 2024-05-06T00:27:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0503e-05 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.875177+0000", 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2646000000000001e-05 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.894287+0000", 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019109206 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.894427+0000", 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014071 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.895143+0000", 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00071604100000000005 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.896969+0000", 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001825478 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.898129+0000", 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001160183 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.898269+0000", 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013992099999999999 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:08.898361+0000", 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2021000000000005e-05 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:559 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.082786+0000", 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.3645441330000008, 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085290509999999993, 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 559 2024-05-06T00:27:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082786+0000", 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082786+0000", 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082788+0000", 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8339999999999999e-06 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082797+0000", 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9789999999999999e-06 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082818+0000", 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0449e-05 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082840+0000", 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2325999999999999e-05 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084697+0000", 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018566310000000001 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084941+0000", 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000244164 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085882+0000", 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00094077400000000002 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.089326+0000", 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034444150000000002 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091071+0000", 2024-05-06T00:27:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001745251 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091209+0000", 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000137403 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091316+0000", 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010682499999999999 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:560 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.082873+0000", 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.3644579829999994, 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011230254, 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 560 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082873+0000", 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082873+0000", 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082874+0000", 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.519e-06 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082881+0000", 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2359999999999998e-06 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082905+0000", 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3723e-05 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.082923+0000", 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7685000000000001e-05 2024-05-06T00:27:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.086807+0000", 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038844249999999999 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.087035+0000", 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022779199999999999 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.087751+0000", 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00071593500000000001 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.093832+0000", 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00608137 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.093872+0000", 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9755000000000001e-05 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.093998+0000", 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012544300000000001 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.094103+0000", 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105371 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:561 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.083118+0000", 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.3642123300000009, 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013481519000000001, 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 561 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083118+0000", 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083118+0000", 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083120+0000", 2024-05-06T00:27:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8330000000000001e-06 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083130+0000", 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6860000000000001e-06 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083157+0000", 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6789000000000001e-05 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083177+0000", 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0781999999999999e-05 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.089474+0000", 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062963079999999996 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.089656+0000", 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000182561 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.090298+0000", 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00064222900000000002 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.094401+0000", 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041023639999999998 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096354+0000", 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001953097 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096484+0000", 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013024600000000001 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096600+0000", 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115624 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:570 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.083568+0000", 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.3637625100000008, 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015897214999999999, 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 570 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083568+0000", 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083568+0000", 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083570+0000", 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083581+0000", 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0392e-05 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083608+0000", 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6783000000000001e-05 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.083627+0000", 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9779000000000002e-05 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091442+0000", 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0078141879999999997 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.091738+0000", 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029659400000000001 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.092803+0000", 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010646270000000001 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096632+0000", 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038287680000000002 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099233+0000", 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026018700000000001 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099370+0000", 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136339 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099465+0000", 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5521999999999998e-05 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:571 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.084855+0000", 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.3624758650000004, 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014876495999999999, 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.543 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 571 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084855+0000", 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084855+0000", 2024-05-06T00:27:17.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084857+0000", 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4329999999999998e-06 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084869+0000", 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1966e-05 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084896+0000", 2024-05-06T00:27:17.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6441999999999999e-05 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.084910+0000", 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4543e-05 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.094566+0000", 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096554580000000004 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.094732+0000", 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016648100000000001 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.095350+0000", 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061754699999999997 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099539+0000", 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041893290000000003 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099574+0000", 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4647999999999998e-05 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099647+0000", 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3570000000000002e-05 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.099731+0000", 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4079000000000006e-05 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.548 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:580 3.f 3:f6d9c2a5:::rbd_object_map.110c6af4f54c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:09.085074+0000", 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.3622562039999995, 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015523696, 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 580 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.549 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085074+0000", 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085074+0000", 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085076+0000", 2024-05-06T00:27:17.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6199999999999999e-06 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085084+0000", 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4540000000000007e-06 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085101+0000", 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6545000000000002e-05 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.085111+0000", 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0189e-05 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096709+0000", 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011597573 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.096885+0000", 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017651099999999999 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.098140+0000", 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001254783 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.100126+0000", 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019856320000000002 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.100454+0000", 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032824299999999998 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.100539+0000", 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4958999999999998e-05 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:09.100598+0000", 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9187000000000001e-05 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:870 3.10 3:09c2179a:::rbd_data.110c6af4f54c.0000000000000001:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.028008+0000", 2024-05-06T00:27:17.554 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.4193230320000003, 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089563750000000008, 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 870 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028008+0000", 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028008+0000", 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028009+0000", 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0920000000000001e-06 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028014+0000", 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6760000000000004e-06 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028030+0000", 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5770999999999999e-05 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028038+0000", 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1880000000000002e-06 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028100+0000", 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1240000000000003e-05 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028169+0000", 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9183999999999994e-05 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028348+0000", 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017878499999999999 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.033228+0000", 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004880087 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.036862+0000", 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036341099999999999 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.036918+0000", 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6029000000000002e-05 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.036964+0000", 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6213e-05 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4737.0:873 3.16 3:6d311e83:::rbd_data.110c6af4f54c.0000000000000003:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.4191487509999998, 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010204229, 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4737", 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 873 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028183+0000", 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4199999999999995e-07 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028185+0000", 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9720000000000001e-06 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028192+0000", 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5119999999999997e-06 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028197+0000", 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6369999999999998e-06 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028254+0000", 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7014e-05 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028337+0000", 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3403000000000001e-05 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.028632+0000", 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029496999999999998 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.032702+0000", 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040701230000000001 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038251+0000", 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005549099 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038332+0000", 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0694999999999999e-05 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:10.038386+0000", 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4162000000000001e-05 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:17.566 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:17.567 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-05-06T00:27:17.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:17.568 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:17.565+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:17.668 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:17.665+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:17.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:17.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:17.698 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:17.698 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:17.711 INFO:teuthology.orchestra.run.smithi186.stdout:1521 read 0x20a5a00 thru 0x20b07c3 (0xadc4 bytes) 2024-05-06T00:27:17.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:17.712 INFO:teuthology.orchestra.run.smithi186.stdout:1522 read 0x25c4000 thru 0x25d0709 (0xc70a bytes) 2024-05-06T00:27:17.763 INFO:teuthology.orchestra.run.smithi186.stdout:1523 write 0x7ba4000 thru 0x7ba6b43 (0x2b44 bytes) 2024-05-06T00:27:17.765 INFO:teuthology.orchestra.run.smithi186.stdout:1524 punch from 0x14b9400 to 0x14c4600, (0xb200 bytes) 2024-05-06T00:27:17.769 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:17.768+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:17.844 INFO:teuthology.orchestra.run.smithi186.stdout:1528 trunc from 0x7ba6c00 to 0xde9c600 2024-05-06T00:27:17.869 INFO:teuthology.orchestra.run.smithi186.stdout:1531 write 0x6fef200 thru 0x6ffb49b (0xc29c bytes) 2024-05-06T00:27:17.870 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:17.869+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:17.870 INFO:teuthology.orchestra.run.smithi186.stdout:1532 write 0x9cf1800 thru 0x9cfb9cc (0xa1cd bytes) 2024-05-06T00:27:17.871 INFO:teuthology.orchestra.run.smithi186.stdout:1533 write 0xd163400 thru 0xd169c8c (0x688d bytes) 2024-05-06T00:27:17.871 INFO:teuthology.orchestra.run.smithi186.stdout:1534 trunc from 0xde9c600 to 0xdc1be00 2024-05-06T00:27:17.890 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:17.890 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:17.890 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:17.890 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:17.890 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:17.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:17.916 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.4 to 1 (10000) 2024-05-06T00:27:17.939 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-05-06T00:27:17.959 INFO:teuthology.orchestra.run.smithi186.stdout:1535 read 0x84f2a00 thru 0x84f5827 (0x2e28 bytes) 2024-05-06T00:27:17.970 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:17.968+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:17.981 INFO:teuthology.orchestra.run.smithi186.stdout:1538 write 0xe8c4000 thru 0xe8c7ea6 (0x3ea7 bytes) 2024-05-06T00:27:17.982 INFO:teuthology.orchestra.run.smithi186.stdout:1540 write 0x9e88a00 thru 0x9e94241 (0xb842 bytes) 2024-05-06T00:27:17.986 INFO:teuthology.orchestra.run.smithi186.stdout:1541 write 0xc620e00 thru 0xc62e734 (0xd935 bytes) 2024-05-06T00:27:18.010 INFO:teuthology.orchestra.run.smithi186.stdout:1542 trunc from 0xe8c8000 to 0x5bb8400 2024-05-06T00:27:18.049 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4412.0:558 3.17 3:e8136097:::rbd_data.110c17d674e9.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write 5419008~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.363838534999999, 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065722799999999998, 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4412", 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1097715279", 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 558 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685114+0000", 2024-05-06T00:27:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685115+0000", 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9399999999999995e-07 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685137+0000", 2024-05-06T00:27:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2628000000000001e-05 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685145+0000", 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6310000000000003e-06 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.685152+0000", 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3889999999999999e-06 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686121+0000", 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096883500000000003 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686199+0000", 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7911000000000007e-05 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-06T00:27:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.686364+0000", 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000164754 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.690386+0000", 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040223309999999996 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691639+0000", 2024-05-06T00:27:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012525640000000001 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691667+0000", 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8084000000000001e-05 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:49.691686+0000", 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9658999999999999e-05 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.244593389999999, 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:27:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:27:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:27:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.244577187000001, 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:27:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2527 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch ping cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.888483237999999, 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16552214100000001, 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2527 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160471+0000", 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.206e-06 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160477+0000", 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7440000000000001e-06 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160491+0000", 2024-05-06T00:27:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3739e-05 2024-05-06T00:27:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160506+0000", 2024-05-06T00:27:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.473e-05 2024-05-06T00:27:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160712+0000", 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020643899999999999 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160736+0000", 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3676000000000002e-05 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325517+0000", 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16478073700000001 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325685+0000", 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016879300000000001 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325992+0000", 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030607699999999998 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2558 3.1f 3:fcc12fb6:::rbd_data.110cc8beeb04.0000000000000000:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.550561141999999, 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067308660000000003, 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2558 2024-05-06T00:27:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498394+0000", 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2469999999999998e-06 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498404+0000", 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0577000000000001e-05 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498435+0000", 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0743e-05 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498452+0000", 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6650000000000002e-05 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498620+0000", 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168202 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498840+0000", 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022057199999999999 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.499360+0000", 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000519149 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502603+0000", 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032430419999999998 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504911+0000", 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023087350000000001 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505025+0000", 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011343 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505122+0000", 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7518999999999999e-05 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1858 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.548061054000001, 2024-05-06T00:27:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069279169999999996, 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1858 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500894+0000", 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.587e-06 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500907+0000", 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.29e-05 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500940+0000", 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2709999999999997e-05 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500961+0000", 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0916000000000001e-05 2024-05-06T00:27:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507119+0000", 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061583690000000003 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507405+0000", 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028573800000000001 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507820+0000", 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000414697 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2571 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.547683599999999, 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079485920000000009, 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2571 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501271+0000", 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3700000000000002e-06 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501290+0000", 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8505999999999999e-05 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501321+0000", 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.099e-05 2024-05-06T00:27:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501340+0000", 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8987e-05 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508595+0000", 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072554910000000002 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508787+0000", 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019116399999999999 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509218+0000", 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043108399999999998 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1859 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.546871960000001, 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078054739999999997, 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1859 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502083+0000", 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7070000000000001e-06 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502097+0000", 2024-05-06T00:27:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3447e-05 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502130+0000", 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3300000000000003e-05 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502151+0000", 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0665999999999998e-05 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509314+0000", 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071635980000000002 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509492+0000", 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000177975 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509886+0000", 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000393781 2024-05-06T00:27:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2572 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.546793395999998, 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085800230000000009, 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2572 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502162+0000", 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.993e-06 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502177+0000", 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.506e-05 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502206+0000", 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8379000000000001e-05 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502226+0000", 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9927000000000001e-05 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510022+0000", 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077960959999999998 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510238+0000", 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021639200000000001 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510739+0000", 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00050117599999999999 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2573 3.12 3:4926f074:::rbd_data.110cc8beeb04.0000000000000006:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.546214968000001, 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066001020000000001, 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2573 2024-05-06T00:27:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502740+0000", 2024-05-06T00:27:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.88e-06 2024-05-06T00:27:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502749+0000", 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8400000000000007e-06 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502771+0000", 2024-05-06T00:27:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1622999999999999e-05 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502788+0000", 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7e-05 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502947+0000", 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015869300000000001 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503160+0000", 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000213291 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503643+0000", 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048288900000000002 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507989+0000", 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043459170000000004 2024-05-06T00:27:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509041+0000", 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010519209999999999 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509218+0000", 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017686899999999999 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509338+0000", 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120179 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2580 3.12 3:4e19e050:::rbd_data.110cc8beeb04.000000000000002d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.546100015, 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079317220000000004, 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2580 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502854+0000", 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502862+0000", 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9140000000000008e-06 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502882+0000", 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9885999999999999e-05 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502896+0000", 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4219999999999999e-05 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505092+0000", 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021958530000000002 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505289+0000", 2024-05-06T00:27:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019717899999999999 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505714+0000", 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042437900000000001 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508626+0000", 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029123479999999999 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510454+0000", 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001828292 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510664+0000", 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000209829 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510784+0000", 2024-05-06T00:27:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120386 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2581 3.1b 3:d89117cf:::rbd_data.110cc8beeb04.0000000000000033:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.546029632, 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010296013, 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2581 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502924+0000", 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.243e-06 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502931+0000", 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502948+0000", 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7374e-05 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502962+0000", 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3653e-05 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503093+0000", 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013094499999999999 2024-05-06T00:27:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503278+0000", 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000185582 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503709+0000", 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043043999999999998 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507476+0000", 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003766901 2024-05-06T00:27:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513089+0000", 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005613743 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513164+0000", 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4276999999999994e-05 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513219+0000", 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5432000000000001e-05 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1860 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.544483868, 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096507989999999998, 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1860 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.027e-06 2024-05-06T00:27:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504488+0000", 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6629999999999998e-05 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504523+0000", 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4178999999999999e-05 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504544+0000", 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1627000000000001e-05 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513785+0000", 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092403420000000003 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513895+0000", 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110796 2024-05-06T00:27:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514120+0000", 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000224198 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2586 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051770034032] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.543630811, 2024-05-06T00:27:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070106719999999999, 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2586 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505323+0000", 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.243e-06 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505330+0000", 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1169999999999998e-06 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505349+0000", 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8482999999999999e-05 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505368+0000", 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9576999999999999e-05 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511761+0000", 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063923230000000001 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511904+0000", 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014312600000000001 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.512333+0000", 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042880300000000001 2024-05-06T00:27:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1862 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.540389575999999, 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006446907, 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1862 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508563+0000", 2024-05-06T00:27:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508565+0000", 2024-05-06T00:27:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508574+0000", 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2199999999999998e-06 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508595+0000", 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0783000000000001e-05 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508610+0000", 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5367e-05 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514622+0000", 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060122470000000001 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514725+0000", 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103009 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515010+0000", 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028462099999999998 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2591 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051769985984] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.539981823000002, 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066352190000000004, 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2591 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508973+0000", 2024-05-06T00:27:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.39e-06 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508983+0000", 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0254e-05 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509010+0000", 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6398999999999999e-05 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509033+0000", 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.298e-05 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515092+0000", 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060590740000000002 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515201+0000", 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108602 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515606+0000", 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040551999999999998 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4737.0:869 3.1f 3:fd5b52f4:::rbd_data.110c6af4f54c.0000000000000000:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:10.026885+0000", 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.0220675519999993, 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079024339999999998, 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4737", 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 869 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026885+0000", 2024-05-06T00:27:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026885+0000", 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026887+0000", 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1670000000000002e-06 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026903+0000", 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5288999999999999e-05 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026924+0000", 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1518999999999998e-05 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026934+0000", 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5219999999999996e-06 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027035+0000", 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010157000000000001 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027170+0000", 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134843 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027437+0000", 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026655400000000001 2024-05-06T00:27:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.031220+0000", 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037831499999999999 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.034669+0000", 2024-05-06T00:27:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003449021 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.034719+0000", 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0659999999999999e-05 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.034788+0000", 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8139000000000006e-05 2024-05-06T00:27:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4737.0:875 3.1b 3:dd6c69eb:::rbd_data.110c6af4f54c.0000000000000004:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:10.027326+0000", 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.0216267850000005, 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010240862999999999, 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4737", 2024-05-06T00:27:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 875 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027326+0000", 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027326+0000", 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027327+0000", 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4139999999999999e-06 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027334+0000", 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6889999999999998e-06 2024-05-06T00:27:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027352+0000", 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8168e-05 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027370+0000", 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8079000000000001e-05 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027449+0000", 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8417000000000005e-05 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027546+0000", 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7602999999999994e-05 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027793+0000", 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024628399999999999 2024-05-06T00:27:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028868+0000", 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010758300000000001 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037454+0000", 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085859209999999998 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037516+0000", 2024-05-06T00:27:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1569000000000006e-05 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037567+0000", 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0889e-05 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4737.0:878 3.1 3:832629b4:::rbd_data.110c6af4f54c.0000000000000005:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:10.027395+0000", 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.0215572730000009, 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010172182, 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4737", 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 878 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027395+0000", 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027395+0000", 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027396+0000", 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027401+0000", 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6539999999999998e-06 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027411+0000", 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0513000000000001e-05 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027418+0000", 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2250000000000003e-06 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027483+0000", 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4262000000000002e-05 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027561+0000", 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8595000000000002e-05 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027749+0000", 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018780899999999999 2024-05-06T00:27:18.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028942+0000", 2024-05-06T00:27:18.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001192654 2024-05-06T00:27:18.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037428+0000", 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084866060000000007 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037489+0000", 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0686000000000001e-05 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037568+0000", 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8559000000000005e-05 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4737.0:889 3.6 3:645a233e:::rbd_data.110c6af4f54c.000000000000000b:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.0207706279999993, 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093640990000000007, 2024-05-06T00:27:18.162 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4737", 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 889 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028183+0000", 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0210000000000001e-06 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028188+0000", 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2870000000000004e-06 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028199+0000", 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0861e-05 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028207+0000", 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2199999999999992e-06 2024-05-06T00:27:18.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028285+0000", 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7979999999999995e-05 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028387+0000", 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010209600000000001 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028589+0000", 2024-05-06T00:27:18.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020201299999999999 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.029814+0000", 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012242500000000001 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037413+0000", 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075991619999999996 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037482+0000", 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8792000000000001e-05 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037546+0000", 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4417000000000003e-05 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.168 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:18.170 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:18.168+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:18.170 INFO:teuthology.orchestra.run.smithi186.stdout:1543 read 0x1cd4a00 thru 0x1ce1dfb (0xd3fc bytes) 2024-05-06T00:27:18.171 INFO:teuthology.orchestra.run.smithi186.stdout:1544 punch from 0x5288200 to 0x5296c00, (0xea00 bytes) 2024-05-06T00:27:18.171 INFO:teuthology.orchestra.run.smithi186.stdout:1545 write 0x24dd200 thru 0x24e539d (0x819e bytes) 2024-05-06T00:27:18.171 INFO:teuthology.orchestra.run.smithi186.stdout:1546 trunc from 0x5bb8400 to 0x5116000 2024-05-06T00:27:18.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:18.203 INFO:teuthology.orchestra.run.smithi186.stdout:1548 read 0x4472600 thru 0x4477ffc (0x59fd bytes) 2024-05-06T00:27:18.205 INFO:teuthology.orchestra.run.smithi186.stdout:1550 read 0x3da4400 thru 0x3da9f94 (0x5b95 bytes) 2024-05-06T00:27:18.245 INFO:teuthology.orchestra.run.smithi186.stdout:1552 write 0xca8ac00 thru 0xca93e18 (0x9219 bytes) 2024-05-06T00:27:18.247 INFO:teuthology.orchestra.run.smithi186.stdout:1553 read 0x7b19800 thru 0x7b25e86 (0xc687 bytes) 2024-05-06T00:27:18.247 INFO:teuthology.orchestra.run.smithi186.stdout:1555 write 0x1ffaa00 thru 0x1ffad08 (0x309 bytes) 2024-05-06T00:27:18.261 INFO:teuthology.orchestra.run.smithi186.stdout:1556 trunc from 0xca94000 to 0x78ed800 2024-05-06T00:27:18.273 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:18.269+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:18.303 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:18.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:18.303 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:18.303 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:18.317 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:18.319 INFO:teuthology.orchestra.run.smithi186.stdout:1557 trunc from 0x78ed800 to 0xcfa6e00 2024-05-06T00:27:18.345 INFO:teuthology.orchestra.run.smithi186.stdout:1559 read 0xa8c8200 thru 0xa8d3426 (0xb227 bytes) 2024-05-06T00:27:18.372 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:18.368+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:18.409 INFO:teuthology.orchestra.run.smithi186.stdout:1560 trunc from 0xcfa6e00 to 0x6d2400 2024-05-06T00:27:18.473 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:18.472+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:18.488 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:18.488 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:18.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:18.522 INFO:teuthology.orchestra.run.smithi186.stdout:1562 read 0x5a6400 thru 0x5b60b0 (0xfcb1 bytes) 2024-05-06T00:27:18.525 INFO:teuthology.orchestra.run.smithi186.stdout:1563 read 0x3a0a00 thru 0x3ac7de (0xbddf bytes) 2024-05-06T00:27:18.539 INFO:teuthology.orchestra.run.smithi186.stdout:1564 trunc from 0x6d2400 to 0x5bce000 2024-05-06T00:27:18.552 INFO:teuthology.orchestra.run.smithi186.stdout:1566 read 0x35d2400 thru 0x35d542b (0x302c bytes) 2024-05-06T00:27:18.554 INFO:teuthology.orchestra.run.smithi186.stdout:1567 write 0x124e800 thru 0x1257f4e (0x974f bytes) 2024-05-06T00:27:18.554 INFO:teuthology.orchestra.run.smithi186.stdout:1568 write 0x436d800 thru 0x4377142 (0x9943 bytes) 2024-05-06T00:27:18.574 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:18.573+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:18.621 INFO:teuthology.orchestra.run.smithi186.stdout:1569 write 0xe9e9200 thru 0xe9f2547 (0x9348 bytes) 2024-05-06T00:27:18.623 INFO:teuthology.orchestra.run.smithi186.stdout:1571 punch from 0x3d54000 to 0x3d5d600, (0x9600 bytes) 2024-05-06T00:27:18.625 INFO:teuthology.orchestra.run.smithi186.stdout:1572 write 0x5970a00 thru 0x597d576 (0xcb77 bytes) 2024-05-06T00:27:18.625 INFO:teuthology.orchestra.run.smithi186.stdout:1573 write 0xbf31800 thru 0xbf39b4c (0x834d bytes) 2024-05-06T00:27:18.632 INFO:teuthology.orchestra.run.smithi186.stdout:1574 trunc from 0xe9f2600 to 0xa99d800 2024-05-06T00:27:18.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:228 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 397312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.034141825999999, 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072743440000000003, 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 228 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597016+0000", 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.443e-06 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597023+0000", 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8340000000000001e-06 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597044+0000", 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0876000000000001e-05 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597055+0000", 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1314e-05 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597176+0000", 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120565 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597192+0000", 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6337e-05 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603743+0000", 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065506820000000004 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603976+0000", 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000233355 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604289+0000", 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031293800000000002 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:229 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 528384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.033983927000001, 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075584060000000002, 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:27:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 229 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597175+0000", 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8929999999999999e-06 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597184+0000", 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2040000000000003e-06 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597200+0000", 2024-05-06T00:27:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5642999999999999e-05 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597209+0000", 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4329999999999993e-06 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597301+0000", 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1588999999999993e-05 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:27:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597309+0000", 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604360+0000", 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070513030000000001 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604516+0000", 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015564999999999999 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604731+0000", 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000215267 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:165 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.947000185, 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010266747, 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 165 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684158+0000", 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6929999999999999e-06 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684170+0000", 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1933000000000001e-05 2024-05-06T00:27:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684198+0000", 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8266999999999999e-05 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684221+0000", 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2736999999999999e-05 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687824+0000", 2024-05-06T00:27:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003602863 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687915+0000", 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0804999999999995e-05 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688264+0000", 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034917100000000001 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693979+0000", 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057148959999999997 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694148+0000", 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168951 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694315+0000", 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000167016 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694423+0000", 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108415 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:298 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.871665534000002, 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089449070000000002, 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 298 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759493+0000", 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8029999999999999e-06 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759503+0000", 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4769999999999992e-06 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759525+0000", 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2067999999999999e-05 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759540+0000", 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5296999999999999e-05 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.762625+0000", 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030855399999999999 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.762723+0000", 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7750000000000004e-05 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763171+0000", 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044755199999999999 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766326+0000", 2024-05-06T00:27:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031554080000000002 2024-05-06T00:27:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768328+0000", 2024-05-06T00:27:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020015530000000001 2024-05-06T00:27:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768387+0000", 2024-05-06T00:27:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9283000000000001e-05 2024-05-06T00:27:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768436+0000", 2024-05-06T00:27:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9175999999999998e-05 2024-05-06T00:27:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:300 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.864743231999999, 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077869669999999997, 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 300 2024-05-06T00:27:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766415+0000", 2024-05-06T00:27:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.536e-06 2024-05-06T00:27:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766423+0000", 2024-05-06T00:27:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2849999999999995e-06 2024-05-06T00:27:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766435+0000", 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2121e-05 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766443+0000", 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9140000000000008e-06 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766490+0000", 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6517999999999997e-05 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766495+0000", 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-06T00:27:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769158+0000", 2024-05-06T00:27:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026631390000000001 2024-05-06T00:27:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769268+0000", 2024-05-06T00:27:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010981599999999999 2024-05-06T00:27:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769603+0000", 2024-05-06T00:27:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033493500000000002 2024-05-06T00:27:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.773467+0000", 2024-05-06T00:27:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038639920000000001 2024-05-06T00:27:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774044+0000", 2024-05-06T00:27:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00057753000000000004 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774139+0000", 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5315999999999993e-05 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774201+0000", 2024-05-06T00:27:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1054000000000002e-05 2024-05-06T00:27:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:324 3.19 3:99a6ca97:::rbd_data.110cc8beeb04.0000000000000013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.840147797, 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062413850000000003, 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 324 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791010+0000", 2024-05-06T00:27:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0449999999999999e-06 2024-05-06T00:27:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791015+0000", 2024-05-06T00:27:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4689999999999996e-06 2024-05-06T00:27:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791030+0000", 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4892000000000001e-05 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791038+0000", 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8460000000000008e-06 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791106+0000", 2024-05-06T00:27:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7994e-05 2024-05-06T00:27:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791190+0000", 2024-05-06T00:27:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3337000000000005e-05 2024-05-06T00:27:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791293+0000", 2024-05-06T00:27:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103757 2024-05-06T00:27:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795405+0000", 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041115479999999996 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797183+0000", 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001778097 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797213+0000", 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.004e-05 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797250+0000", 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7360000000000001e-05 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:328 3.19 3:9fbbd60a:::rbd_data.110cc8beeb04.000000000000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.839999485, 2024-05-06T00:27:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063090830000000001, 2024-05-06T00:27:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 328 2024-05-06T00:27:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791158+0000", 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7000000000000003e-07 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791165+0000", 2024-05-06T00:27:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3220000000000004e-06 2024-05-06T00:27:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791178+0000", 2024-05-06T00:27:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2078e-05 2024-05-06T00:27:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791184+0000", 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791609+0000", 2024-05-06T00:27:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042469600000000001 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791782+0000", 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017316400000000001 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.792020+0000", 2024-05-06T00:27:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023781599999999999 2024-05-06T00:27:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795867+0000", 2024-05-06T00:27:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038475010000000001 2024-05-06T00:27:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797389+0000", 2024-05-06T00:27:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015214339999999999 2024-05-06T00:27:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797418+0000", 2024-05-06T00:27:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8977999999999999e-05 2024-05-06T00:27:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797466+0000", 2024-05-06T00:27:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8489999999999998e-05 2024-05-06T00:27:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1015 3.8 e29/18 3:1391aa96:::rbd_data.110cc8beeb04.0000000000000025:head v 29'6, mlcod=29'4)", 2024-05-06T00:27:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.377094379999999, 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075649460000000003, 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254063+0000", 2024-05-06T00:27:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4099999999999997e-07 2024-05-06T00:27:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254072+0000", 2024-05-06T00:27:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6370000000000005e-06 2024-05-06T00:27:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254084+0000", 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1725999999999999e-05 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254091+0000", 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.379e-06 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254155+0000", 2024-05-06T00:27:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3906999999999996e-05 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254177+0000", 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1526e-05 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261558+0000", 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073814409999999999 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261627+0000", 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9388999999999996e-05 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1017 3.7 e29/18 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head v 29'11, mlcod=29'9)", 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.377049392, 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075152739999999997, 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254108+0000", 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254112+0000", 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254119+0000", 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7800000000000003e-06 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254124+0000", 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8509999999999998e-06 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254189+0000", 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4757000000000004e-05 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254209+0000", 2024-05-06T00:27:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0307000000000001e-05 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261566+0000", 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007356596 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261623+0000", 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7111000000000003e-05 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1022 3.8 e29/18 3:17fe3906:::rbd_data.110cc8beeb04.0000000000000036:head v 29'7, mlcod=29'4)", 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.376887276000001, 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073889450000000001, 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254270+0000", 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254275+0000", 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6120000000000001e-06 2024-05-06T00:27:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254283+0000", 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2199999999999992e-06 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254289+0000", 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9880000000000001e-06 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254360+0000", 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1062000000000005e-05 2024-05-06T00:27:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254379+0000", 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9194e-05 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261632+0000", 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00725255 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261658+0000", 2024-05-06T00:27:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6857000000000001e-05 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1018 3.7 e29/18 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head v 29'12, mlcod=29'9)", 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.376835449000001, 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073213799999999997, 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254322+0000", 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5400000000000002e-07 2024-05-06T00:27:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254326+0000", 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5979999999999999e-06 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254334+0000", 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1489999999999997e-06 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254339+0000", 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1399999999999999e-06 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254387+0000", 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8102000000000001e-05 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254403+0000", 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5982e-05 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261626+0000", 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072235140000000003 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261643+0000", 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6441000000000001e-05 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2574 3.4 3:218edb7b:::rbd_data.110cc8beeb04.0000000000000007:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.128320033000001, 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006796074, 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2574 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7329999999999999e-06 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502852+0000", 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2203e-05 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502884+0000", 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2543000000000003e-05 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502906+0000", 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2027000000000001e-05 2024-05-06T00:27:18.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503066+0000", 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159946 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503260+0000", 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019375100000000001 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503688+0000", 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042773399999999998 2024-05-06T00:27:18.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.506368+0000", 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002680689 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509377+0000", 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003008667 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509527+0000", 2024-05-06T00:27:18.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149498 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509633+0000", 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1279 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.970042792999999, 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084436479999999998, 2024-05-06T00:27:18.719 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1279 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661117+0000", 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6929999999999999e-06 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661132+0000", 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5153999999999999e-05 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661170+0000", 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7886000000000003e-05 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661188+0000", 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7816e-05 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661295+0000", 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001072 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661597+0000", 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030253000000000002 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.662320+0000", 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072273200000000002 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669339+0000", 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070190219999999998 2024-05-06T00:27:18.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669377+0000", 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7935999999999997e-05 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669495+0000", 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117599 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669558+0000", 2024-05-06T00:27:18.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3079999999999999e-05 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1293 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.967735233999999, 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066635549999999998, 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1293 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663424+0000", 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6599999999999999e-06 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663439+0000", 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4633e-05 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663471+0000", 2024-05-06T00:27:18.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1986999999999997e-05 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663491+0000", 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0435999999999999e-05 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663643+0000", 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151929 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663863+0000", 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021996200000000001 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.664839+0000", 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00097599500000000001 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669583+0000", 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047435070000000001 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669694+0000", 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011181899999999999 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669963+0000", 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026898800000000001 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670085+0000", 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012163900000000001 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1294 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.967435310000001, 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086615830000000005, 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1294 2024-05-06T00:27:18.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663724+0000", 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.61e-06 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663739+0000", 2024-05-06T00:27:18.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.447e-05 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663768+0000", 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9957000000000001e-05 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663789+0000", 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0420000000000001e-05 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.667197+0000", 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003407963 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.667372+0000", 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000175532 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.668276+0000", 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00090397299999999997 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670141+0000", 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001864239 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672093+0000", 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019525650000000001 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672271+0000", 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017802900000000001 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672383+0000", 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011182499999999999 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1303 3.4 3:214da2ae:::rbd_data.110c7776112a.0000000000000007:head [set-alloc-hint object_size 33554432 write_size 33554432,write 2125824~23552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.961165079000001, 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063999010000000004, 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1303 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:18.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669994+0000", 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8169999999999999e-06 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670033+0000", 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9113000000000003e-05 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670057+0000", 2024-05-06T00:27:18.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4334000000000001e-05 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670078+0000", 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1539000000000002e-05 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672106+0000", 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00202797 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672301+0000", 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019495800000000001 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672680+0000", 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037862600000000002 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.675864+0000", 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003183745 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676177+0000", 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031343499999999998 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676313+0000", 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013609700000000001 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676392+0000", 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8267000000000001e-05 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2707 3.4 3:21d3ac4d:::rbd_data.110cd55b09e5.0000000000000064:head [set-alloc-hint object_size 2097152 write_size 2097152,write 929280~9216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:13.119026+0000", 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5121312160000002, 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006225291, 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2707 2024-05-06T00:27:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.119026+0000", 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.119026+0000", 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.119027+0000", 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.238e-06 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.119040+0000", 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3696000000000001e-05 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.119055+0000", 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4559000000000001e-05 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.119063+0000", 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3829999999999992e-06 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.119156+0000", 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2998000000000006e-05 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.119316+0000", 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015953199999999999 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.120102+0000", 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00078567999999999997 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.122790+0000", 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026886969999999999 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.125057+0000", 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022665250000000001 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.125171+0000", 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011464 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.125251+0000", 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9343000000000003e-05 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4646.0:329 3.2 3:46f4a6be:::rbd_header.110cc6eb3028:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-05-06T00:27:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:14.745609+0000", 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.8855476319999998, 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061865770000000004, 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4646", 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/4090392681", 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 329 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745609+0000", 2024-05-06T00:27:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745609+0000", 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745610+0000", 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8299999999999996e-07 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745613+0000", 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3840000000000001e-06 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745621+0000", 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5800000000000003e-06 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745626+0000", 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2499999999999997e-06 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.749135+0000", 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003508826 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.749189+0000", 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4279e-05 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.751712+0000", 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025227190000000001 2024-05-06T00:27:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.751746+0000", 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4087000000000001e-05 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.751796+0000", 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9768999999999997e-05 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4646.0:330 3.2 3:46f4a6be:::rbd_header.110cc6eb3028:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:14.745823+0000", 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.885333642, 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062150410000000001, 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4646", 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/4090392681", 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 330 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745823+0000", 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745823+0000", 2024-05-06T00:27:18.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745824+0000", 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8299999999999995e-07 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745828+0000", 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3900000000000003e-06 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745839+0000", 2024-05-06T00:27:18.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0896e-05 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.745847+0000", 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3590000000000002e-06 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.749276+0000", 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034292319999999999 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.749306+0000", 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9893e-05 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.751942+0000", 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026356460000000002 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.751974+0000", 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2809999999999999e-05 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.752038+0000", 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3831999999999995e-05 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:4289 3.2 3:46f4a6be:::rbd_header.110cc6eb3028:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:14.746796+0000", 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.8843606340000001, 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061689329999999997, 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:18.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4289 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.746796+0000", 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.746796+0000", 2024-05-06T00:27:18.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.746797+0000", 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2399999999999996e-07 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.746802+0000", 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4900000000000002e-06 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.746848+0000", 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6113999999999999e-05 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.746854+0000", 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8669999999999999e-06 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.749700+0000", 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002846964 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.749730+0000", 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9961e-05 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.752851+0000", 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031201829999999999 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.752908+0000", 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7840999999999997e-05 2024-05-06T00:27:18.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.752965+0000", 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6589000000000003e-05 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:18.764 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:18.766 INFO:teuthology.orchestra.run.smithi186.stdout:1575 write 0x63ce800 thru 0x63d68fc (0x80fd bytes) 2024-05-06T00:27:18.766 INFO:teuthology.orchestra.run.smithi186.stdout:1576 write 0x160fc00 thru 0x1616bdb (0x6fdc bytes) 2024-05-06T00:27:18.766 INFO:teuthology.orchestra.run.smithi186.stdout:1577 write 0xc99cc00 thru 0xc9a2ae2 (0x5ee3 bytes) 2024-05-06T00:27:18.766 INFO:teuthology.orchestra.run.smithi186.stdout:1579 write 0x645a200 thru 0x646566a (0xb46b bytes) 2024-05-06T00:27:18.766 INFO:teuthology.orchestra.run.smithi186.stdout:1583 write 0x22d9000 thru 0x22e3fcb (0xafcc bytes) 2024-05-06T00:27:18.767 INFO:teuthology.orchestra.run.smithi186.stdout:1584 trunc from 0xc9a2c00 to 0xe3bb800 2024-05-06T00:27:18.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:18.767 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:18.764+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:18.802 INFO:teuthology.orchestra.run.smithi186.stdout:1585 write 0x65cd000 thru 0x65d19fe (0x49ff bytes) 2024-05-06T00:27:18.824 INFO:teuthology.orchestra.run.smithi186.stdout:1586 trunc from 0xe3bb800 to 0x6a9f000 2024-05-06T00:27:18.856 INFO:teuthology.orchestra.run.smithi186.stdout:1587 read 0x4f19000 thru 0x4f1ee87 (0x5e88 bytes) 2024-05-06T00:27:18.857 INFO:teuthology.orchestra.run.smithi186.stdout:1589 write 0x66b1000 thru 0x66b91e7 (0x81e8 bytes) 2024-05-06T00:27:18.867 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:18.865+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:18.887 INFO:teuthology.orchestra.run.smithi186.stdout:1590 write 0x94b4800 thru 0x94c0fe8 (0xc7e9 bytes) 2024-05-06T00:27:18.923 INFO:teuthology.orchestra.run.smithi186.stdout:1591 write 0x9bd7200 thru 0x9be0009 (0x8e0a bytes) 2024-05-06T00:27:18.925 INFO:teuthology.orchestra.run.smithi186.stdout:1592 read 0x937ae00 thru 0x937e28c (0x348d bytes) 2024-05-06T00:27:18.927 INFO:teuthology.orchestra.run.smithi186.stdout:1593 punch from 0x8985c00 to 0x8993200, (0xd600 bytes) 2024-05-06T00:27:18.934 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.5 to 1 (10000) 2024-05-06T00:27:18.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:27:18.958 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:18.958 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:18.958 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:18.958 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:18.960 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-05-06T00:27:18.961 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:27:18.969 INFO:teuthology.orchestra.run.smithi186.stdout:1595 write 0xa867800 thru 0xa86e1a2 (0x69a3 bytes) 2024-05-06T00:27:18.971 INFO:teuthology.orchestra.run.smithi186.stdout:1596 write 0x2360a00 thru 0x2367595 (0x6b96 bytes) 2024-05-06T00:27:18.973 INFO:teuthology.orchestra.run.smithi186.stdout:1597 read 0x7697000 thru 0x76a2478 (0xb479 bytes) 2024-05-06T00:27:18.973 INFO:teuthology.orchestra.run.smithi186.stdout:1599 write 0x75f0c00 thru 0x75f7d4b (0x714c bytes) 2024-05-06T00:27:18.973 INFO:teuthology.orchestra.run.smithi186.stdout:1600 punch from 0x35c4600 to 0x35ce600, (0xa000 bytes) 2024-05-06T00:27:18.975 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:18.975 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:18.973+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:18.976 INFO:teuthology.orchestra.run.smithi186.stdout:1602 read 0x989f600 thru 0x98a77e0 (0x81e1 bytes) 2024-05-06T00:27:19.004 INFO:teuthology.orchestra.run.smithi186.stdout:1603 write 0x8264a00 thru 0x8269621 (0x4c22 bytes) 2024-05-06T00:27:19.037 INFO:teuthology.orchestra.run.smithi186.stdout:1604 write 0xe44d600 thru 0xe45a0ef (0xcaf0 bytes) 2024-05-06T00:27:19.058 INFO:teuthology.orchestra.run.smithi186.stdout:1605 trunc from 0xe45a200 to 0x3056000 2024-05-06T00:27:19.068 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:19.068+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:19.154 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:19.154 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:19.154 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:19.154 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:19.154 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:19.166 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:19.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:19.171 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:19.168+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:19.179 INFO:teuthology.orchestra.run.smithi186.stdout:1607 write 0xe215400 thru 0xe223c4a (0xe84b bytes) 2024-05-06T00:27:19.180 INFO:teuthology.orchestra.run.smithi186.stdout:1608 read 0xb10aa00 thru 0xb11a3f8 (0xf9f9 bytes) 2024-05-06T00:27:19.210 INFO:teuthology.orchestra.run.smithi186.stdout:1609 trunc from 0xe223e00 to 0x194f800 2024-05-06T00:27:19.270 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:19.268+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:19.273 INFO:teuthology.orchestra.run.smithi186.stdout:1610 read 0x75be00 thru 0x75da44 (0x1c45 bytes) 2024-05-06T00:27:19.276 INFO:teuthology.orchestra.run.smithi186.stdout:1612 read 0x17a6400 thru 0x17b109f (0xaca0 bytes) 2024-05-06T00:27:19.277 INFO:teuthology.orchestra.run.smithi186.stdout:1613 read 0x10cbe00 thru 0x10ccba9 (0xdaa bytes) 2024-05-06T00:27:19.278 INFO:teuthology.orchestra.run.smithi186.stdout:1615 punch from 0x449800 to 0x44b800, (0x2000 bytes) 2024-05-06T00:27:19.279 INFO:teuthology.orchestra.run.smithi186.stdout:1616 read 0x18fb600 thru 0x1903e64 (0x8865 bytes) 2024-05-06T00:27:19.284 INFO:teuthology.orchestra.run.smithi186.stdout:1617 trunc from 0x194f800 to 0x3cec00 2024-05-06T00:27:19.313 INFO:teuthology.orchestra.run.smithi186.stdout:1619 read 0x35be00 thru 0x35e892 (0x2a93 bytes) 2024-05-06T00:27:19.316 INFO:teuthology.orchestra.run.smithi186.stdout:1622 read 0x2b3200 thru 0x2bad37 (0x7b38 bytes) 2024-05-06T00:27:19.317 INFO:teuthology.orchestra.run.smithi186.stdout:1625 read 0x10bc00 thru 0x114cf1 (0x90f2 bytes) 2024-05-06T00:27:19.318 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:19.318 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:19.338 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:19.338 INFO:teuthology.orchestra.run.smithi186.stdout:1627 write 0x317be00 thru 0x31876e5 (0xb8e6 bytes) 2024-05-06T00:27:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2567 3.1e 3:7ff04bbe:::rbd_data.110cc8beeb04.0000000000000005:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.844106416999999, 2024-05-06T00:27:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091482830000000001, 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2567 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498657+0000", 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.75e-06 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498666+0000", 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3030000000000005e-06 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498690+0000", 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-05 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498707+0000", 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7326999999999999e-05 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501155+0000", 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002447884 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501468+0000", 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031281100000000002 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501983+0000", 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051484999999999996 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505388+0000", 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034055240000000001 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507550+0000", 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021622690000000001 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507694+0000", 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014377899999999999 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507804+0000", 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109486 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1779 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.6866463710000001, 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095752939999999998, 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1779 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656116+0000", 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9699999999999995e-07 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656120+0000", 2024-05-06T00:27:19.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5939999999999998e-06 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656128+0000", 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0309999999999998e-06 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656133+0000", 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.659748+0000", 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003615231 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.659987+0000", 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023881900000000001 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.660658+0000", 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067093800000000005 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.663069+0000", 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024110109999999998 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665388+0000", 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023186579999999999 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665608+0000", 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022027 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665691+0000", 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2698e-05 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1841 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.6573087900000001, 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011157446999999999, 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1841 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685454+0000", 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2200000000000002e-07 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685459+0000", 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4820000000000002e-06 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685469+0000", 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8020000000000003e-06 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685476+0000", 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-06 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.689403+0000", 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003927346 2024-05-06T00:27:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.689582+0000", 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017826300000000001 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.690411+0000", 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00082928899999999998 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696488+0000", 2024-05-06T00:27:19.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060775960000000002 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696529+0000", 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0416000000000001e-05 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696571+0000", 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1931000000000002e-05 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696610+0000", 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9660000000000003e-05 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1848 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.6558234829999998, 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011575561999999999, 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1848 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686939+0000", 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.136e-06 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686946+0000", 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2790000000000004e-06 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686959+0000", 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3774000000000001e-05 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686969+0000", 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2070000000000001e-06 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.691625+0000", 2024-05-06T00:27:19.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046563450000000001 2024-05-06T00:27:19.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.691714+0000", 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8651000000000002e-05 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692727+0000", 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010135400000000001 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696654+0000", 2024-05-06T00:27:19.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039264529999999999 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698290+0000", 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016358880000000001 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698414+0000", 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000124082 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698514+0000", 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100207 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1849 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.6556093579999995, 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013152664, 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1849 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:19.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687153+0000", 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6599999999999994e-07 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687172+0000", 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8868999999999998e-05 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687187+0000", 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4491999999999999e-05 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687196+0000", 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6039999999999999e-06 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.694179+0000", 2024-05-06T00:27:19.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069822119999999998 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.694396+0000", 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000217856 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.695435+0000", 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001038349 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700028+0000", 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045936090000000002 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700067+0000", 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9048999999999998e-05 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700203+0000", 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013551700000000001 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700305+0000", 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102141 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1850 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.6549791950000001, 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01443141, 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1850 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687784+0000", 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.311e-06 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687789+0000", 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5289999999999999e-06 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687809+0000", 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9911e-05 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687818+0000", 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9549999999999998e-06 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696755+0000", 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089369069999999991 2024-05-06T00:27:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696977+0000", 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000221337 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.697535+0000", 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00055843500000000003 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700358+0000", 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028226850000000001 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702017+0000", 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001659537 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702117+0000", 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9984999999999999e-05 2024-05-06T00:27:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702214+0000", 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6817999999999994e-05 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1869 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.6505536090000001, 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011357313000000001, 2024-05-06T00:27:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1869 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692210+0000", 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.514e-06 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692219+0000", 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2399999999999996e-06 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692240+0000", 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1537000000000001e-05 2024-05-06T00:27:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692257+0000", 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6670000000000001e-05 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698589+0000", 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063314920000000002 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698673+0000", 2024-05-06T00:27:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4358999999999996e-05 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.699065+0000", 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039182699999999997 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702254+0000", 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031890780000000001 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703473+0000", 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012188730000000001 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703526+0000", 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3588999999999998e-05 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703565+0000", 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9134000000000002e-05 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.386 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1870 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.6504496040000003, 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011641416999999999, 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.387 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1870 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692314+0000", 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.42e-06 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692323+0000", 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1409999999999994e-06 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692346+0000", 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2955999999999998e-05 2024-05-06T00:27:19.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692363+0000", 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6923999999999999e-05 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700432+0000", 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080692569999999998 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700598+0000", 2024-05-06T00:27:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166422 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.701369+0000", 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00077104099999999998 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703579+0000", 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022098 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703712+0000", 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133378 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703863+0000", 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015091700000000001 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703954+0000", 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0161000000000006e-05 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2082 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.5287203370000002, 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098291899999999998, 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2082 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814043+0000", 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2720000000000001e-06 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814049+0000", 2024-05-06T00:27:19.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814061+0000", 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1815e-05 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814069+0000", 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4630000000000001e-06 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.816625+0000", 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025561239999999999 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.816767+0000", 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014136199999999999 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.817683+0000", 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091655999999999997 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823412+0000", 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057287509999999998 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823578+0000", 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016635 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823763+0000", 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184172 2024-05-06T00:27:19.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823871+0000", 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108068 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2083 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.5286257010000002, 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099999149999999998, 2024-05-06T00:27:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2083 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814137+0000", 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9299999999999997e-07 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814141+0000", 2024-05-06T00:27:19.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5750000000000002e-06 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814151+0000", 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4450000000000002e-06 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814160+0000", 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3279999999999995e-06 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.818686+0000", 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045255809999999999 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.818863+0000", 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017779199999999999 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.819847+0000", 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098399199999999994 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823546+0000", 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036989969999999999 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823898+0000", 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035146899999999998 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824030+0000", 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000132392 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824136+0000", 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105651 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2084 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.5285383069999998, 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011218071, 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.405 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2084 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814225+0000", 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.13e-06 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814230+0000", 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1150000000000002e-06 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814243+0000", 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3101999999999999e-05 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814250+0000", 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8759999999999999e-06 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.821215+0000", 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069655439999999997 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.821467+0000", 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025156400000000002 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.822255+0000", 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00078842500000000004 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824161+0000", 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019054429999999999 2024-05-06T00:27:19.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825286+0000", 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001125687 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825375+0000", 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9013999999999999e-05 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825442+0000", 2024-05-06T00:27:19.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6171000000000001e-05 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2733 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:13.129521+0000", 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.2132408950000002, 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086158499999999996, 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.411 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2733 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129521+0000", 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129521+0000", 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129522+0000", 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0100000000000003e-07 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129526+0000", 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3549999999999998e-06 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129536+0000", 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7059999999999999e-06 2024-05-06T00:27:19.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129543+0000", 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7170000000000002e-06 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.132475+0000", 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029324469999999999 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.132542+0000", 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7098e-05 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.132810+0000", 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026769700000000001 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.137696+0000", 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048860309999999999 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138049+0000", 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000352992 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138096+0000", 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7265999999999998e-05 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138137+0000", 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0639999999999997e-05 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.416 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2734 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:13.129580+0000", 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.2131817060000003, 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086660130000000002, 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.417 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2734 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129580+0000", 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129580+0000", 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129581+0000", 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129585+0000", 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129594+0000", 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0950000000000002e-06 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129603+0000", 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4610000000000008e-06 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.133428+0000", 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003825742 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.133498+0000", 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.97e-05 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.133794+0000", 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000296256 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.137768+0000", 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039736629999999997 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138147+0000", 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037913399999999999 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138209+0000", 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2342999999999996e-05 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138246+0000", 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6704e-05 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2735 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.422 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:13.129644+0000", 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.2131182369999998, 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086873039999999999, 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2735 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129644+0000", 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129644+0000", 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129644+0000", 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7499999999999999e-07 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129649+0000", 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.464e-06 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129660+0000", 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1124000000000001e-05 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129669+0000", 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7860000000000002e-06 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.134386+0000", 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004717316 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.134505+0000", 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011895 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.135142+0000", 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063672399999999997 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.137979+0000", 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028371210000000002 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138253+0000", 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027441899999999998 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138297+0000", 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3788000000000002e-05 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138331+0000", 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3837000000000001e-05 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2736 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:13.129779+0000", 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.2129823350000004, 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086912199999999995, 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.428 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2736 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129779+0000", 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129779+0000", 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129780+0000", 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1100000000000005e-07 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129785+0000", 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5859999999999998e-06 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129808+0000", 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3376000000000001e-05 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.129816+0000", 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9780000000000005e-06 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.135739+0000", 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059226890000000001 2024-05-06T00:27:19.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.135803+0000", 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4282999999999994e-05 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.136066+0000", 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026321100000000001 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138036+0000", 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001969925 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138338+0000", 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030165500000000001 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138424+0000", 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5915999999999995e-05 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:13.138471+0000", 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6789999999999998e-05 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4646.0:104 3.8 3:11ca53b4:::rbd_object_map.110cc6eb3028:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:14.555257+0000", 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7875049509999998, 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085634600000000002, 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4646", 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/4090392681", 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 104 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.555257+0000", 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.555257+0000", 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.555258+0000", 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8000000000000004e-07 2024-05-06T00:27:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.555264+0000", 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.857e-06 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.555275+0000", 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1372999999999999e-05 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.555284+0000", 2024-05-06T00:27:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6270000000000006e-06 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.559013+0000", 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037291680000000002 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.559115+0000", 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102067 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.559411+0000", 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029585800000000003 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.562453+0000", 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003042796 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.563676+0000", 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012226089999999999 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.563761+0000", 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4543999999999996e-05 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.563820+0000", 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9681e-05 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4646.0:109 3.8 3:11ca53b4:::rbd_object_map.110cc6eb3028:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-05-06T00:27:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.785423711, 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010787722, 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4646", 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/4090392681", 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 109 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557341+0000", 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.027e-06 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557357+0000", 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5926999999999998e-05 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557392+0000", 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5264e-05 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557414+0000", 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1557e-05 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.561705+0000", 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042916580000000003 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.561768+0000", 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2188999999999997e-05 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.562103+0000", 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033499199999999999 2024-05-06T00:27:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.567706+0000", 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056037730000000003 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568002+0000", 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029544 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568076+0000", 2024-05-06T00:27:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3796999999999998e-05 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568126+0000", 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0098e-05 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4863.0:3614 3.d e43/18 3:b17d50d3:::rbd_object_map.110cf57807c6:head v 43'210, mlcod=43'205)", 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.765118+0000", 2024-05-06T00:27:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.57764336100000002, 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0087561240000000005, 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765118+0000", 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765118+0000", 2024-05-06T00:27:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765126+0000", 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-06 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765144+0000", 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8331000000000001e-05 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765171+0000", 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6420000000000001e-05 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765189+0000", 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7827000000000001e-05 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.766291+0000", 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001102223 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.766346+0000", 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5050999999999999e-05 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.773757+0000", 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074109479999999997 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.773875+0000", 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117564 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4863.0:3623 3.d e43/18 3:b17d50d3:::rbd_object_map.110cf57807c6:head v 43'218, mlcod=43'208)", 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.56832649999999996, 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097994039999999994, 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774437+0000", 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774450+0000", 2024-05-06T00:27:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3257e-05 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774476+0000", 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6108000000000001e-05 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774496+0000", 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9843e-05 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.777187+0000", 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026909 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.777241+0000", 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4104000000000002e-05 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784162+0000", 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069210519999999996 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784235+0000", 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2596999999999996e-05 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4863.0:3625 3.d e43/18 3:b17d50d3:::rbd_object_map.110cf57807c6:head v 43'219, mlcod=43'208)", 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.775265+0000", 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.56749680000000002, 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090135049999999998, 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775265+0000", 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775265+0000", 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:19.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775267+0000", 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.584e-06 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775284+0000", 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7283999999999999e-05 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775315+0000", 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1192999999999998e-05 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775334+0000", 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8507000000000001e-05 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.778537+0000", 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032031659999999999 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.778617+0000", 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0464999999999996e-05 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784238+0000", 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056206149999999998 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784278+0000", 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0691000000000001e-05 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:19.459 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:19.461 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:19.461 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:19.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1628 write 0x96b4e00 thru 0x96c3b5b (0xed5c bytes) 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1629 read 0x18d8600 thru 0x18db08c (0x2a8d bytes) 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1630 punch from 0x2c83400 to 0x2c8c600, (0x9200 bytes) 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1631 read 0x2380400 thru 0x238220c (0x1e0d bytes) 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1633 punch from 0x66baa00 to 0x66c7800, (0xce00 bytes) 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1636 write 0x8a03800 thru 0x8a03937 (0x138 bytes) 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1638 read 0x175cc00 thru 0x17646da (0x7adb bytes) 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1639 punch from 0x2511c00 to 0x2515600, (0x3a00 bytes) 2024-05-06T00:27:19.464 INFO:teuthology.orchestra.run.smithi186.stdout:1640 trunc from 0x96c3c00 to 0x7c57000 2024-05-06T00:27:19.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:27:19.465 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:27:19.467 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-05-06T00:27:18.950804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":296948447,"num_objects":195,"num_object_clones":47,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":45,"num_read":36682,"num_read_kb":75311,"num_write":3243,"num_write_kb":37096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3004,"ondisk_log_size":3004,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":200528,"kb_used_data":9280,"kb_used_omap":9,"kb_used_meta":191158,"kb_avail":566030512,"statfs":{"total":579820584960,"available":579615244288,"internally_reserved":0,"allocated":9502720,"data_stored":7615953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":195746464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1854075,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8795,"num_read_kb":21248,"num_write":917,"num_write_kb":4735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039886"},"pg_stats":[{"pgid":"3.1f","version":"41'37","reported_seq":2324,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921365+0000","last_change":"2024-05-06T00:26:45.712771+0000","last_active":"2024-05-06T00:27:16.921365+0000","last_peered":"2024-05-06T00:27:16.921365+0000","last_clean":"2024-05-06T00:27:16.921365+0000","last_became_active":"2024-05-06T00:26:43.694494+0000","last_became_peered":"2024-05-06T00:26:43.694494+0000","last_unstale":"2024-05-06T00:27:16.921365+0000","last_undegraded":"2024-05-06T00:27:16.921365+0000","last_fullsized":"2024-05-06T00:27:16.921365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:33:11.948130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":1969718,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":2243,"num_read_kb":10913,"num_write":34,"num_write_kb":463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"41'45","reported_seq":625,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920467+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:27:16.920467+0000","last_peered":"2024-05-06T00:27:16.920467+0000","last_clean":"2024-05-06T00:27:16.920467+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:27:16.920467+0000","last_undegraded":"2024-05-06T00:27:16.920467+0000","last_fullsized":"2024-05-06T00:27:16.920467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:24:23.294646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":24933888,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":532,"num_read_kb":834,"num_write":41,"num_write_kb":503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"41'20","reported_seq":300,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932802+0000","last_change":"2024-05-06T00:26:45.733037+0000","last_active":"2024-05-06T00:27:16.932802+0000","last_peered":"2024-05-06T00:27:16.932802+0000","last_clean":"2024-05-06T00:27:16.932802+0000","last_became_active":"2024-05-06T00:26:43.711475+0000","last_became_peered":"2024-05-06T00:26:43.711475+0000","last_unstale":"2024-05-06T00:27:16.932802+0000","last_undegraded":"2024-05-06T00:27:16.932802+0000","last_fullsized":"2024-05-06T00:27:16.932802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:30:15.531802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":5783552,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":211,"num_read_kb":435,"num_write":19,"num_write_kb":318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"42'41","reported_seq":1065,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.346087+0000","last_change":"2024-05-06T00:26:45.732216+0000","last_active":"2024-05-06T00:27:17.346087+0000","last_peered":"2024-05-06T00:27:17.346087+0000","last_clean":"2024-05-06T00:27:17.346087+0000","last_became_active":"2024-05-06T00:26:43.715832+0000","last_became_peered":"2024-05-06T00:26:43.715832+0000","last_unstale":"2024-05-06T00:27:17.346087+0000","last_undegraded":"2024-05-06T00:27:17.346087+0000","last_fullsized":"2024-05-06T00:27:17.346087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:21:20.643715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":1716224,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1053,"num_read_kb":3850,"num_write":32,"num_write_kb":217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"40'227","reported_seq":1874,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921140+0000","last_change":"2024-05-06T00:26:45.713563+0000","last_active":"2024-05-06T00:27:16.921140+0000","last_peered":"2024-05-06T00:27:16.921140+0000","last_clean":"2024-05-06T00:27:16.921140+0000","last_became_active":"2024-05-06T00:26:43.707253+0000","last_became_peered":"2024-05-06T00:26:43.707253+0000","last_unstale":"2024-05-06T00:27:16.921140+0000","last_undegraded":"2024-05-06T00:27:16.921140+0000","last_fullsized":"2024-05-06T00:27:16.921140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:48:00.643161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":26897443,"num_objects":14,"num_object_clones":5,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":5,"num_read":2964,"num_read_kb":5247,"num_write":236,"num_write_kb":2917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"41'61","reported_seq":1529,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.466358+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:27:17.466358+0000","last_peered":"2024-05-06T00:27:17.466358+0000","last_clean":"2024-05-06T00:27:17.466358+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:27:17.466358+0000","last_undegraded":"2024-05-06T00:27:17.466358+0000","last_fullsized":"2024-05-06T00:27:17.466358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:49:12.084974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":12849765,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":1856,"num_read_kb":2020,"num_write":47,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"41'89","reported_seq":964,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927789+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:27:16.927789+0000","last_peered":"2024-05-06T00:27:16.927789+0000","last_clean":"2024-05-06T00:27:16.927789+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:27:16.927789+0000","last_undegraded":"2024-05-06T00:27:16.927789+0000","last_fullsized":"2024-05-06T00:27:16.927789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:32:46.511073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":6083584,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1507,"num_read_kb":1551,"num_write":65,"num_write_kb":556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"41'30","reported_seq":306,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.411390+0000","last_change":"2024-05-06T00:26:45.713666+0000","last_active":"2024-05-06T00:27:17.411390+0000","last_peered":"2024-05-06T00:27:17.411390+0000","last_clean":"2024-05-06T00:27:17.411390+0000","last_became_active":"2024-05-06T00:26:43.688284+0000","last_became_peered":"2024-05-06T00:26:43.688284+0000","last_unstale":"2024-05-06T00:27:17.411390+0000","last_undegraded":"2024-05-06T00:27:17.411390+0000","last_fullsized":"2024-05-06T00:27:17.411390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:09:57.844336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":131072,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":210,"num_read_kb":2527,"num_write":28,"num_write_kb":327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"40'73","reported_seq":553,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921210+0000","last_change":"2024-05-06T00:26:45.713245+0000","last_active":"2024-05-06T00:27:16.921210+0000","last_peered":"2024-05-06T00:27:16.921210+0000","last_clean":"2024-05-06T00:27:16.921210+0000","last_became_active":"2024-05-06T00:26:43.698443+0000","last_became_peered":"2024-05-06T00:26:43.698443+0000","last_unstale":"2024-05-06T00:27:16.921210+0000","last_undegraded":"2024-05-06T00:27:16.921210+0000","last_fullsized":"2024-05-06T00:27:16.921210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:01:52.941891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":33026090,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":3,"num_read":487,"num_read_kb":1464,"num_write":70,"num_write_kb":2753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"41'30","reported_seq":364,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932549+0000","last_change":"2024-05-06T00:26:45.733718+0000","last_active":"2024-05-06T00:27:16.932549+0000","last_peered":"2024-05-06T00:27:16.932549+0000","last_clean":"2024-05-06T00:27:16.932549+0000","last_became_active":"2024-05-06T00:26:43.703465+0000","last_became_peered":"2024-05-06T00:26:43.703465+0000","last_unstale":"2024-05-06T00:27:16.932549+0000","last_undegraded":"2024-05-06T00:27:16.932549+0000","last_fullsized":"2024-05-06T00:27:16.932549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:58:49.184957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":27495424,"num_objects":10,"num_object_clones":4,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":275,"num_read_kb":406,"num_write":30,"num_write_kb":303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"41'56","reported_seq":401,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.409835+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:27:17.409835+0000","last_peered":"2024-05-06T00:27:17.409835+0000","last_clean":"2024-05-06T00:27:17.409835+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:27:17.409835+0000","last_undegraded":"2024-05-06T00:27:17.409835+0000","last_fullsized":"2024-05-06T00:27:17.409835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:15:29.843480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4943360,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":4,"num_read":331,"num_read_kb":675,"num_write":56,"num_write_kb":11828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"38'44","reported_seq":285,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921272+0000","last_change":"2024-05-06T00:26:45.710285+0000","last_active":"2024-05-06T00:27:16.921272+0000","last_peered":"2024-05-06T00:27:16.921272+0000","last_clean":"2024-05-06T00:27:16.921272+0000","last_became_active":"2024-05-06T00:26:43.706306+0000","last_became_peered":"2024-05-06T00:26:43.706306+0000","last_unstale":"2024-05-06T00:27:16.921272+0000","last_undegraded":"2024-05-06T00:27:16.921272+0000","last_fullsized":"2024-05-06T00:27:16.921272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:11:56.623563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":10747920,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":197,"num_read_kb":421,"num_write":36,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921365+0000","last_change":"2024-05-06T00:26:36.150377+0000","last_active":"2024-05-06T00:27:16.921365+0000","last_peered":"2024-05-06T00:27:16.921365+0000","last_clean":"2024-05-06T00:27:16.921365+0000","last_became_active":"2024-05-06T00:26:36.149496+0000","last_became_peered":"2024-05-06T00:26:36.149496+0000","last_unstale":"2024-05-06T00:27:16.921365+0000","last_undegraded":"2024-05-06T00:27:16.921365+0000","last_fullsized":"2024-05-06T00:27:16.921365+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:54.410549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"40'336","reported_seq":1977,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921196+0000","last_change":"2024-05-06T00:27:01.825846+0000","last_active":"2024-05-06T00:27:16.921196+0000","last_peered":"2024-05-06T00:27:16.921196+0000","last_clean":"2024-05-06T00:27:16.921196+0000","last_became_active":"2024-05-06T00:27:01.825382+0000","last_became_peered":"2024-05-06T00:27:01.825382+0000","last_unstale":"2024-05-06T00:27:16.921196+0000","last_undegraded":"2024-05-06T00:27:16.921196+0000","last_fullsized":"2024-05-06T00:27:16.921196+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:02:41.081020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":262226,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":3969,"num_read_kb":3364,"num_write":408,"num_write_kb":705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927600+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:27:16.927600+0000","last_peered":"2024-05-06T00:27:16.927600+0000","last_clean":"2024-05-06T00:27:16.927600+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:27:16.927600+0000","last_undegraded":"2024-05-06T00:27:16.927600+0000","last_fullsized":"2024-05-06T00:27:16.927600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"41'278","reported_seq":1119,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920184+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:27:16.920184+0000","last_peered":"2024-05-06T00:27:16.920184+0000","last_clean":"2024-05-06T00:27:16.920184+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:27:16.920184+0000","last_undegraded":"2024-05-06T00:27:16.920184+0000","last_fullsized":"2024-05-06T00:27:16.920184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:53:08.579738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":1048613,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2544,"num_read_kb":2252,"num_write":376,"num_write_kb":735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921398+0000","last_change":"2024-05-06T00:26:35.642448+0000","last_active":"2024-05-06T00:27:16.921398+0000","last_peered":"2024-05-06T00:27:16.921398+0000","last_clean":"2024-05-06T00:27:16.921398+0000","last_became_active":"2024-05-06T00:26:35.641678+0000","last_became_peered":"2024-05-06T00:26:35.641678+0000","last_unstale":"2024-05-06T00:27:16.921398+0000","last_undegraded":"2024-05-06T00:27:16.921398+0000","last_fullsized":"2024-05-06T00:27:16.921398+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:32:32.988296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"41'232","reported_seq":758,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927325+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:27:16.927325+0000","last_peered":"2024-05-06T00:27:16.927325+0000","last_clean":"2024-05-06T00:27:16.927325+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:27:16.927325+0000","last_undegraded":"2024-05-06T00:27:16.927325+0000","last_fullsized":"2024-05-06T00:27:16.927325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:31:49.483428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":16210501,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":1333,"num_read_kb":2586,"num_write":340,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927464+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:27:16.927464+0000","last_peered":"2024-05-06T00:27:16.927464+0000","last_clean":"2024-05-06T00:27:16.927464+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:27:16.927464+0000","last_undegraded":"2024-05-06T00:27:16.927464+0000","last_fullsized":"2024-05-06T00:27:16.927464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"41'33","reported_seq":148,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932985+0000","last_change":"2024-05-06T00:26:45.733467+0000","last_active":"2024-05-06T00:27:16.932985+0000","last_peered":"2024-05-06T00:27:16.932985+0000","last_clean":"2024-05-06T00:27:16.932985+0000","last_became_active":"2024-05-06T00:26:43.703416+0000","last_became_peered":"2024-05-06T00:26:43.703416+0000","last_unstale":"2024-05-06T00:27:16.932985+0000","last_undegraded":"2024-05-06T00:27:16.932985+0000","last_fullsized":"2024-05-06T00:27:16.932985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:39:19.838424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":32,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":93,"num_write":33,"num_write_kb":396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"41'33","reported_seq":192,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.181160+0000","last_change":"2024-05-06T00:26:45.720211+0000","last_active":"2024-05-06T00:27:17.181160+0000","last_peered":"2024-05-06T00:27:17.181160+0000","last_clean":"2024-05-06T00:27:17.181160+0000","last_became_active":"2024-05-06T00:26:43.705235+0000","last_became_peered":"2024-05-06T00:26:43.705235+0000","last_unstale":"2024-05-06T00:27:17.181160+0000","last_undegraded":"2024-05-06T00:27:17.181160+0000","last_fullsized":"2024-05-06T00:27:17.181160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:50:23.495101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":375845,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":100,"num_read_kb":361,"num_write":32,"num_write_kb":347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932398+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:27:16.932398+0000","last_peered":"2024-05-06T00:27:16.932398+0000","last_clean":"2024-05-06T00:27:16.932398+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:27:16.932398+0000","last_undegraded":"2024-05-06T00:27:16.932398+0000","last_fullsized":"2024-05-06T00:27:16.932398+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"40'145","reported_seq":356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927117+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:27:16.927117+0000","last_peered":"2024-05-06T00:27:16.927117+0000","last_clean":"2024-05-06T00:27:16.927117+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:27:16.927117+0000","last_undegraded":"2024-05-06T00:27:16.927117+0000","last_fullsized":"2024-05-06T00:27:16.927117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:00:32.019838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":5349480,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":664,"num_read_kb":748,"num_write":211,"num_write_kb":797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920086+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:27:16.920086+0000","last_peered":"2024-05-06T00:27:16.920086+0000","last_clean":"2024-05-06T00:27:16.920086+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:27:16.920086+0000","last_undegraded":"2024-05-06T00:27:16.920086+0000","last_fullsized":"2024-05-06T00:27:16.920086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"41'49","reported_seq":440,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920774+0000","last_change":"2024-05-06T00:26:45.712444+0000","last_active":"2024-05-06T00:27:16.920774+0000","last_peered":"2024-05-06T00:27:16.920774+0000","last_clean":"2024-05-06T00:27:16.920774+0000","last_became_active":"2024-05-06T00:26:43.708328+0000","last_became_peered":"2024-05-06T00:26:43.708328+0000","last_unstale":"2024-05-06T00:27:16.920774+0000","last_undegraded":"2024-05-06T00:27:16.920774+0000","last_fullsized":"2024-05-06T00:27:16.920774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:40.650848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":21321728,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":338,"num_read_kb":2220,"num_write":50,"num_write_kb":778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920786+0000","last_change":"2024-05-06T00:26:35.652664+0000","last_active":"2024-05-06T00:27:16.920786+0000","last_peered":"2024-05-06T00:27:16.920786+0000","last_clean":"2024-05-06T00:27:16.920786+0000","last_became_active":"2024-05-06T00:26:35.652017+0000","last_became_peered":"2024-05-06T00:26:35.652017+0000","last_unstale":"2024-05-06T00:27:16.920786+0000","last_undegraded":"2024-05-06T00:27:16.920786+0000","last_fullsized":"2024-05-06T00:27:16.920786+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:34.935271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"42'139","reported_seq":1856,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.339070+0000","last_change":"2024-05-06T00:26:45.725541+0000","last_active":"2024-05-06T00:27:17.339070+0000","last_peered":"2024-05-06T00:27:17.339070+0000","last_clean":"2024-05-06T00:27:17.339070+0000","last_became_active":"2024-05-06T00:26:43.696938+0000","last_became_peered":"2024-05-06T00:26:43.696938+0000","last_unstale":"2024-05-06T00:27:17.339070+0000","last_undegraded":"2024-05-06T00:27:17.339070+0000","last_fullsized":"2024-05-06T00:27:17.339070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:12:11.039209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":58,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2457,"num_read_kb":2220,"num_write":88,"num_write_kb":649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932378+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:27:16.932378+0000","last_peered":"2024-05-06T00:27:16.932378+0000","last_clean":"2024-05-06T00:27:16.932378+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:27:16.932378+0000","last_undegraded":"2024-05-06T00:27:16.932378+0000","last_fullsized":"2024-05-06T00:27:16.932378+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":158,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920821+0000","last_change":"2024-05-06T00:26:30.239639+0000","last_active":"2024-05-06T00:27:16.920821+0000","last_peered":"2024-05-06T00:27:16.920821+0000","last_clean":"2024-05-06T00:27:16.920821+0000","last_became_active":"2024-05-06T00:26:30.237552+0000","last_became_peered":"2024-05-06T00:26:30.237552+0000","last_unstale":"2024-05-06T00:27:16.920821+0000","last_undegraded":"2024-05-06T00:27:16.920821+0000","last_fullsized":"2024-05-06T00:27:16.920821+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:44:57.011856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"42'204","reported_seq":659,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.424149+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:27:17.424149+0000","last_peered":"2024-05-06T00:27:17.424149+0000","last_clean":"2024-05-06T00:27:17.424149+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:27:17.424149+0000","last_undegraded":"2024-05-06T00:27:17.424149+0000","last_fullsized":"2024-05-06T00:27:17.424149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:40:59.218132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":18216088,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":1253,"num_read_kb":1534,"num_write":323,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"41'42","reported_seq":311,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.944800+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:27:16.944800+0000","last_peered":"2024-05-06T00:27:16.944800+0000","last_clean":"2024-05-06T00:27:16.944800+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:27:16.944800+0000","last_undegraded":"2024-05-06T00:27:16.944800+0000","last_fullsized":"2024-05-06T00:27:16.944800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:22:48.068013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":7234048,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":208,"num_read_kb":397,"num_write":40,"num_write_kb":689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"42'197","reported_seq":2191,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.470006+0000","last_change":"2024-05-06T00:26:45.710256+0000","last_active":"2024-05-06T00:27:17.470006+0000","last_peered":"2024-05-06T00:27:17.470006+0000","last_clean":"2024-05-06T00:27:17.470006+0000","last_became_active":"2024-05-06T00:26:43.710572+0000","last_became_peered":"2024-05-06T00:26:43.710572+0000","last_unstale":"2024-05-06T00:27:17.470006+0000","last_undegraded":"2024-05-06T00:27:17.470006+0000","last_fullsized":"2024-05-06T00:27:17.470006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:03:48.514905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":7607843,"num_objects":12,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":3483,"num_read_kb":4838,"num_write":121,"num_write_kb":772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"42'26","reported_seq":165,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.254079+0000","last_change":"2024-05-06T00:26:45.725952+0000","last_active":"2024-05-06T00:27:17.254079+0000","last_peered":"2024-05-06T00:27:17.254079+0000","last_clean":"2024-05-06T00:27:17.254079+0000","last_became_active":"2024-05-06T00:26:43.686980+0000","last_became_peered":"2024-05-06T00:26:43.686980+0000","last_unstale":"2024-05-06T00:27:17.254079+0000","last_undegraded":"2024-05-06T00:27:17.254079+0000","last_fullsized":"2024-05-06T00:27:17.254079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:57:59.257502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":152,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":431,"num_write":25,"num_write_kb":3405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"41'35","reported_seq":392,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.417499+0000","last_change":"2024-05-06T00:26:45.732742+0000","last_active":"2024-05-06T00:27:17.417499+0000","last_peered":"2024-05-06T00:27:17.417499+0000","last_clean":"2024-05-06T00:27:17.417499+0000","last_became_active":"2024-05-06T00:26:43.700115+0000","last_became_peered":"2024-05-06T00:26:43.700115+0000","last_unstale":"2024-05-06T00:27:17.417499+0000","last_undegraded":"2024-05-06T00:27:17.417499+0000","last_fullsized":"2024-05-06T00:27:17.417499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:39:22.920835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":2727936,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":289,"num_read_kb":763,"num_write":35,"num_write_kb":611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"42'23","reported_seq":187,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.418366+0000","last_change":"2024-05-06T00:26:45.711599+0000","last_active":"2024-05-06T00:27:17.418366+0000","last_peered":"2024-05-06T00:27:17.418366+0000","last_clean":"2024-05-06T00:27:17.418366+0000","last_became_active":"2024-05-06T00:26:43.692092+0000","last_became_peered":"2024-05-06T00:26:43.692092+0000","last_unstale":"2024-05-06T00:27:17.418366+0000","last_undegraded":"2024-05-06T00:27:17.418366+0000","last_fullsized":"2024-05-06T00:27:17.418366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:05.056345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":628833,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":105,"num_read_kb":1860,"num_write":21,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"41'52","reported_seq":1121,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.402504+0000","last_change":"2024-05-06T00:26:45.734276+0000","last_active":"2024-05-06T00:27:17.402504+0000","last_peered":"2024-05-06T00:27:17.402504+0000","last_clean":"2024-05-06T00:27:17.402504+0000","last_became_active":"2024-05-06T00:26:43.711491+0000","last_became_peered":"2024-05-06T00:26:43.711491+0000","last_unstale":"2024-05-06T00:27:17.402504+0000","last_undegraded":"2024-05-06T00:27:17.402504+0000","last_fullsized":"2024-05-06T00:27:17.402504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:00:11.880195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":9990144,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1552,"num_read_kb":4490,"num_write":32,"num_write_kb":244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"41'62","reported_seq":174,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.458593+0000","last_change":"2024-05-06T00:26:45.734362+0000","last_active":"2024-05-06T00:27:17.458593+0000","last_peered":"2024-05-06T00:27:17.458593+0000","last_clean":"2024-05-06T00:27:17.458593+0000","last_became_active":"2024-05-06T00:26:43.703972+0000","last_became_peered":"2024-05-06T00:26:43.703972+0000","last_unstale":"2024-05-06T00:27:17.458593+0000","last_undegraded":"2024-05-06T00:27:17.458593+0000","last_fullsized":"2024-05-06T00:27:17.458593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:19:16.880866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":96,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":289,"num_read_kb":258,"num_write":93,"num_write_kb":321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"41'71","reported_seq":831,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932297+0000","last_change":"2024-05-06T00:26:45.732061+0000","last_active":"2024-05-06T00:27:16.932297+0000","last_peered":"2024-05-06T00:27:16.932297+0000","last_clean":"2024-05-06T00:27:16.932297+0000","last_became_active":"2024-05-06T00:26:43.699883+0000","last_became_peered":"2024-05-06T00:26:43.699883+0000","last_unstale":"2024-05-06T00:27:16.932297+0000","last_undegraded":"2024-05-06T00:27:16.932297+0000","last_fullsized":"2024-05-06T00:27:16.932297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:42:16.152037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":8608768,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":1132,"num_read_kb":2537,"num_write":54,"num_write_kb":436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"41'19","reported_seq":185,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.403953+0000","last_change":"2024-05-06T00:26:45.710994+0000","last_active":"2024-05-06T00:27:17.403953+0000","last_peered":"2024-05-06T00:27:17.403953+0000","last_clean":"2024-05-06T00:27:17.403953+0000","last_became_active":"2024-05-06T00:26:43.710745+0000","last_became_peered":"2024-05-06T00:26:43.710745+0000","last_unstale":"2024-05-06T00:27:17.403953+0000","last_undegraded":"2024-05-06T00:27:17.403953+0000","last_fullsized":"2024-05-06T00:27:17.403953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:38:42.599417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":9511936,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":101,"num_read_kb":2011,"num_write":17,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"41'53","reported_seq":488,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920929+0000","last_change":"2024-05-06T00:26:45.713363+0000","last_active":"2024-05-06T00:27:16.920929+0000","last_peered":"2024-05-06T00:27:16.920929+0000","last_clean":"2024-05-06T00:27:16.920929+0000","last_became_active":"2024-05-06T00:26:43.692971+0000","last_became_peered":"2024-05-06T00:26:43.692971+0000","last_unstale":"2024-05-06T00:27:16.920929+0000","last_undegraded":"2024-05-06T00:27:16.920929+0000","last_fullsized":"2024-05-06T00:27:16.920929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:22:06.588605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":19036672,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":3,"num_read":382,"num_read_kb":754,"num_write":49,"num_write_kb":833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"42'150","reported_seq":3066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.317501+0000","last_change":"2024-05-06T00:26:45.711743+0000","last_active":"2024-05-06T00:27:17.317501+0000","last_peered":"2024-05-06T00:27:17.317501+0000","last_clean":"2024-05-06T00:27:17.317501+0000","last_became_active":"2024-05-06T00:26:43.700214+0000","last_became_peered":"2024-05-06T00:26:43.700214+0000","last_unstale":"2024-05-06T00:27:17.317501+0000","last_undegraded":"2024-05-06T00:27:17.317501+0000","last_fullsized":"2024-05-06T00:27:17.317501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:36:59.965708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":11649040,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4402,"num_read_kb":11196,"num_write":100,"num_write_kb":652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":296358079,"num_objects":193,"num_object_clones":47,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":45,"num_read":36616,"num_read_kb":75256,"num_write":3142,"num_write_kb":35468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5529600,"data_stored":5352830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2932,"ondisk_log_size":2932,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640271,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1343488,"data_stored":1080626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640269,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1544192,"data_stored":1196912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640269,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33440,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338400,"statfs":{"total":96636764160,"available":96602521600,"internally_reserved":0,"allocated":2310144,"data_stored":2076056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339640,"statfs":{"total":96636764160,"available":96603791360,"internally_reserved":0,"allocated":1048576,"data_stored":707625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640271,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36456,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335384,"statfs":{"total":96636764160,"available":96599433216,"internally_reserved":0,"allocated":1146880,"data_stored":788703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640269,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33248,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338592,"statfs":{"total":96636764160,"available":96602718208,"internally_reserved":0,"allocated":2109440,"data_stored":1766031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":999826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":607393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":526315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1249280,"data_stored":1304378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1077248,"data_stored":1015602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":876544,"data_stored":899316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:27:19.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:27:19.469 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:19.469 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:19.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:19.470 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:19.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:19.471 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:19.471 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:19.457+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:19.472 INFO:teuthology.orchestra.run.smithi186.stdout:1641 punch from 0x1779800 to 0x1780800, (0x7000 bytes) 2024-05-06T00:27:19.472 INFO:teuthology.orchestra.run.smithi186.stdout:1643 read 0x608a600 thru 0x6090085 (0x5a86 bytes) 2024-05-06T00:27:19.493 INFO:teuthology.orchestra.run.smithi186.stdout:1644 trunc from 0x7c57000 to 0x2ff6000 2024-05-06T00:27:19.547 INFO:teuthology.orchestra.run.smithi186.stdout:1645 trunc from 0x2ff6000 to 0xd93b400 2024-05-06T00:27:19.560 INFO:teuthology.orchestra.run.smithi186.stdout:1646 read 0x8ed8600 thru 0x8edc948 (0x4349 bytes) 2024-05-06T00:27:19.561 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:19.560+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:19.562 INFO:teuthology.orchestra.run.smithi186.stdout:1647 write 0xd7e0e00 thru 0xd7ecd3e (0xbf3f bytes) 2024-05-06T00:27:19.592 INFO:teuthology.orchestra.run.smithi186.stdout:1649 trunc from 0xd93b400 to 0x1932600 2024-05-06T00:27:19.603 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:19.604 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:19.623 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:19.664 INFO:teuthology.orchestra.run.smithi186.stdout:1651 trunc from 0x1932600 to 0xc33dc00 2024-05-06T00:27:19.665 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:19.664+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:19.675 INFO:teuthology.orchestra.run.smithi186.stdout:1652 read 0xae98800 thru 0xae9e767 (0x5f68 bytes) 2024-05-06T00:27:19.677 INFO:teuthology.orchestra.run.smithi186.stdout:1654 punch from 0x673ea00 to 0x6748800, (0x9e00 bytes) 2024-05-06T00:27:19.678 INFO:teuthology.orchestra.run.smithi186.stdout:1655 read 0xab9c800 thru 0xaba3384 (0x6b85 bytes) 2024-05-06T00:27:19.678 INFO:teuthology.orchestra.run.smithi186.stdout:1656 punch from 0xb0e1e00 to 0xb0e9a00, (0x7c00 bytes) 2024-05-06T00:27:19.679 INFO:teuthology.orchestra.run.smithi186.stdout:1657 read 0x6f3aa00 thru 0x6f432e8 (0x88e9 bytes) 2024-05-06T00:27:19.679 INFO:teuthology.orchestra.run.smithi186.stdout:1658 write 0xf00000 thru 0xf0a330 (0xa331 bytes) 2024-05-06T00:27:19.679 INFO:teuthology.orchestra.run.smithi186.stdout:1659 write 0x64e0400 thru 0x64e3f9c (0x3b9d bytes) 2024-05-06T00:27:19.687 INFO:teuthology.orchestra.run.smithi186.stdout:1660 trunc from 0xc33dc00 to 0x8141e00 2024-05-06T00:27:19.742 INFO:teuthology.orchestra.run.smithi186.stdout:1663 trunc from 0x8141e00 to 0xac58a00 2024-05-06T00:27:20.248 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:27:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:27:20.250 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:20.250 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:20.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:20.251 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:20.244+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1665 read 0xd43800 thru 0xd441d7 (0x9d8 bytes) 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1668 read 0xabafc00 thru 0xabbefbc (0xf3bd bytes) 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1670 write 0x2ba9a00 thru 0x2baf15b (0x575c bytes) 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1672 trunc from 0xac58a00 to 0x7bb9c00 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1675 read 0x3095200 thru 0x30979bf (0x27c0 bytes) 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1676 write 0xb74f400 thru 0xb75d1a2 (0xdda3 bytes) 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1678 trunc from 0xb75d200 to 0x515e200 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1679 trunc from 0x515e200 to 0x8507400 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1680 write 0xafe1400 thru 0xafef73e (0xe33f bytes) 2024-05-06T00:27:20.252 INFO:teuthology.orchestra.run.smithi186.stdout:1684 clone 11 order 19 su 131072 sc 10 2024-05-06T00:27:20.253 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-05-06T00:27:18.950804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":296948447,"num_objects":195,"num_object_clones":47,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":45,"num_read":36682,"num_read_kb":75311,"num_write":3243,"num_write_kb":37096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3004,"ondisk_log_size":3004,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":200528,"kb_used_data":9280,"kb_used_omap":9,"kb_used_meta":191158,"kb_avail":566030512,"statfs":{"total":579820584960,"available":579615244288,"internally_reserved":0,"allocated":9502720,"data_stored":7615953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":195746464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1854075,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8795,"num_read_kb":21248,"num_write":917,"num_write_kb":4735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039886"},"pg_stats":[{"pgid":"3.1f","version":"41'37","reported_seq":2324,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921365+0000","last_change":"2024-05-06T00:26:45.712771+0000","last_active":"2024-05-06T00:27:16.921365+0000","last_peered":"2024-05-06T00:27:16.921365+0000","last_clean":"2024-05-06T00:27:16.921365+0000","last_became_active":"2024-05-06T00:26:43.694494+0000","last_became_peered":"2024-05-06T00:26:43.694494+0000","last_unstale":"2024-05-06T00:27:16.921365+0000","last_undegraded":"2024-05-06T00:27:16.921365+0000","last_fullsized":"2024-05-06T00:27:16.921365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:33:11.948130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":1969718,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":2243,"num_read_kb":10913,"num_write":34,"num_write_kb":463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"41'45","reported_seq":625,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920467+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:27:16.920467+0000","last_peered":"2024-05-06T00:27:16.920467+0000","last_clean":"2024-05-06T00:27:16.920467+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:27:16.920467+0000","last_undegraded":"2024-05-06T00:27:16.920467+0000","last_fullsized":"2024-05-06T00:27:16.920467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:24:23.294646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":24933888,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":532,"num_read_kb":834,"num_write":41,"num_write_kb":503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"41'20","reported_seq":300,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932802+0000","last_change":"2024-05-06T00:26:45.733037+0000","last_active":"2024-05-06T00:27:16.932802+0000","last_peered":"2024-05-06T00:27:16.932802+0000","last_clean":"2024-05-06T00:27:16.932802+0000","last_became_active":"2024-05-06T00:26:43.711475+0000","last_became_peered":"2024-05-06T00:26:43.711475+0000","last_unstale":"2024-05-06T00:27:16.932802+0000","last_undegraded":"2024-05-06T00:27:16.932802+0000","last_fullsized":"2024-05-06T00:27:16.932802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:30:15.531802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":5783552,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":211,"num_read_kb":435,"num_write":19,"num_write_kb":318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"42'41","reported_seq":1065,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.346087+0000","last_change":"2024-05-06T00:26:45.732216+0000","last_active":"2024-05-06T00:27:17.346087+0000","last_peered":"2024-05-06T00:27:17.346087+0000","last_clean":"2024-05-06T00:27:17.346087+0000","last_became_active":"2024-05-06T00:26:43.715832+0000","last_became_peered":"2024-05-06T00:26:43.715832+0000","last_unstale":"2024-05-06T00:27:17.346087+0000","last_undegraded":"2024-05-06T00:27:17.346087+0000","last_fullsized":"2024-05-06T00:27:17.346087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:21:20.643715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":1716224,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1053,"num_read_kb":3850,"num_write":32,"num_write_kb":217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"40'227","reported_seq":1874,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921140+0000","last_change":"2024-05-06T00:26:45.713563+0000","last_active":"2024-05-06T00:27:16.921140+0000","last_peered":"2024-05-06T00:27:16.921140+0000","last_clean":"2024-05-06T00:27:16.921140+0000","last_became_active":"2024-05-06T00:26:43.707253+0000","last_became_peered":"2024-05-06T00:26:43.707253+0000","last_unstale":"2024-05-06T00:27:16.921140+0000","last_undegraded":"2024-05-06T00:27:16.921140+0000","last_fullsized":"2024-05-06T00:27:16.921140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:48:00.643161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":26897443,"num_objects":14,"num_object_clones":5,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":5,"num_read":2964,"num_read_kb":5247,"num_write":236,"num_write_kb":2917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"41'61","reported_seq":1529,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.466358+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:27:17.466358+0000","last_peered":"2024-05-06T00:27:17.466358+0000","last_clean":"2024-05-06T00:27:17.466358+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:27:17.466358+0000","last_undegraded":"2024-05-06T00:27:17.466358+0000","last_fullsized":"2024-05-06T00:27:17.466358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:49:12.084974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":12849765,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":1856,"num_read_kb":2020,"num_write":47,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"41'89","reported_seq":964,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927789+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:27:16.927789+0000","last_peered":"2024-05-06T00:27:16.927789+0000","last_clean":"2024-05-06T00:27:16.927789+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:27:16.927789+0000","last_undegraded":"2024-05-06T00:27:16.927789+0000","last_fullsized":"2024-05-06T00:27:16.927789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:32:46.511073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":6083584,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1507,"num_read_kb":1551,"num_write":65,"num_write_kb":556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"41'30","reported_seq":306,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.411390+0000","last_change":"2024-05-06T00:26:45.713666+0000","last_active":"2024-05-06T00:27:17.411390+0000","last_peered":"2024-05-06T00:27:17.411390+0000","last_clean":"2024-05-06T00:27:17.411390+0000","last_became_active":"2024-05-06T00:26:43.688284+0000","last_became_peered":"2024-05-06T00:26:43.688284+0000","last_unstale":"2024-05-06T00:27:17.411390+0000","last_undegraded":"2024-05-06T00:27:17.411390+0000","last_fullsized":"2024-05-06T00:27:17.411390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:09:57.844336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":131072,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":210,"num_read_kb":2527,"num_write":28,"num_write_kb":327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"40'73","reported_seq":553,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921210+0000","last_change":"2024-05-06T00:26:45.713245+0000","last_active":"2024-05-06T00:27:16.921210+0000","last_peered":"2024-05-06T00:27:16.921210+0000","last_clean":"2024-05-06T00:27:16.921210+0000","last_became_active":"2024-05-06T00:26:43.698443+0000","last_became_peered":"2024-05-06T00:26:43.698443+0000","last_unstale":"2024-05-06T00:27:16.921210+0000","last_undegraded":"2024-05-06T00:27:16.921210+0000","last_fullsized":"2024-05-06T00:27:16.921210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:01:52.941891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":33026090,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":3,"num_read":487,"num_read_kb":1464,"num_write":70,"num_write_kb":2753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"41'30","reported_seq":364,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932549+0000","last_change":"2024-05-06T00:26:45.733718+0000","last_active":"2024-05-06T00:27:16.932549+0000","last_peered":"2024-05-06T00:27:16.932549+0000","last_clean":"2024-05-06T00:27:16.932549+0000","last_became_active":"2024-05-06T00:26:43.703465+0000","last_became_peered":"2024-05-06T00:26:43.703465+0000","last_unstale":"2024-05-06T00:27:16.932549+0000","last_undegraded":"2024-05-06T00:27:16.932549+0000","last_fullsized":"2024-05-06T00:27:16.932549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:58:49.184957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":27495424,"num_objects":10,"num_object_clones":4,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":275,"num_read_kb":406,"num_write":30,"num_write_kb":303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"41'56","reported_seq":401,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.409835+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:27:17.409835+0000","last_peered":"2024-05-06T00:27:17.409835+0000","last_clean":"2024-05-06T00:27:17.409835+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:27:17.409835+0000","last_undegraded":"2024-05-06T00:27:17.409835+0000","last_fullsized":"2024-05-06T00:27:17.409835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:15:29.843480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4943360,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":4,"num_read":331,"num_read_kb":675,"num_write":56,"num_write_kb":11828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"38'44","reported_seq":285,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921272+0000","last_change":"2024-05-06T00:26:45.710285+0000","last_active":"2024-05-06T00:27:16.921272+0000","last_peered":"2024-05-06T00:27:16.921272+0000","last_clean":"2024-05-06T00:27:16.921272+0000","last_became_active":"2024-05-06T00:26:43.706306+0000","last_became_peered":"2024-05-06T00:26:43.706306+0000","last_unstale":"2024-05-06T00:27:16.921272+0000","last_undegraded":"2024-05-06T00:27:16.921272+0000","last_fullsized":"2024-05-06T00:27:16.921272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:11:56.623563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":10747920,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":197,"num_read_kb":421,"num_write":36,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921365+0000","last_change":"2024-05-06T00:26:36.150377+0000","last_active":"2024-05-06T00:27:16.921365+0000","last_peered":"2024-05-06T00:27:16.921365+0000","last_clean":"2024-05-06T00:27:16.921365+0000","last_became_active":"2024-05-06T00:26:36.149496+0000","last_became_peered":"2024-05-06T00:26:36.149496+0000","last_unstale":"2024-05-06T00:27:16.921365+0000","last_undegraded":"2024-05-06T00:27:16.921365+0000","last_fullsized":"2024-05-06T00:27:16.921365+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:54.410549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"40'336","reported_seq":1977,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921196+0000","last_change":"2024-05-06T00:27:01.825846+0000","last_active":"2024-05-06T00:27:16.921196+0000","last_peered":"2024-05-06T00:27:16.921196+0000","last_clean":"2024-05-06T00:27:16.921196+0000","last_became_active":"2024-05-06T00:27:01.825382+0000","last_became_peered":"2024-05-06T00:27:01.825382+0000","last_unstale":"2024-05-06T00:27:16.921196+0000","last_undegraded":"2024-05-06T00:27:16.921196+0000","last_fullsized":"2024-05-06T00:27:16.921196+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:02:41.081020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":262226,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":3969,"num_read_kb":3364,"num_write":408,"num_write_kb":705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927600+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:27:16.927600+0000","last_peered":"2024-05-06T00:27:16.927600+0000","last_clean":"2024-05-06T00:27:16.927600+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:27:16.927600+0000","last_undegraded":"2024-05-06T00:27:16.927600+0000","last_fullsized":"2024-05-06T00:27:16.927600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"41'278","reported_seq":1119,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920184+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:27:16.920184+0000","last_peered":"2024-05-06T00:27:16.920184+0000","last_clean":"2024-05-06T00:27:16.920184+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:27:16.920184+0000","last_undegraded":"2024-05-06T00:27:16.920184+0000","last_fullsized":"2024-05-06T00:27:16.920184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:53:08.579738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":1048613,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2544,"num_read_kb":2252,"num_write":376,"num_write_kb":735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.921398+0000","last_change":"2024-05-06T00:26:35.642448+0000","last_active":"2024-05-06T00:27:16.921398+0000","last_peered":"2024-05-06T00:27:16.921398+0000","last_clean":"2024-05-06T00:27:16.921398+0000","last_became_active":"2024-05-06T00:26:35.641678+0000","last_became_peered":"2024-05-06T00:26:35.641678+0000","last_unstale":"2024-05-06T00:27:16.921398+0000","last_undegraded":"2024-05-06T00:27:16.921398+0000","last_fullsized":"2024-05-06T00:27:16.921398+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:32:32.988296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"41'232","reported_seq":758,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927325+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:27:16.927325+0000","last_peered":"2024-05-06T00:27:16.927325+0000","last_clean":"2024-05-06T00:27:16.927325+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:27:16.927325+0000","last_undegraded":"2024-05-06T00:27:16.927325+0000","last_fullsized":"2024-05-06T00:27:16.927325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:31:49.483428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":16210501,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":1333,"num_read_kb":2586,"num_write":340,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927464+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:27:16.927464+0000","last_peered":"2024-05-06T00:27:16.927464+0000","last_clean":"2024-05-06T00:27:16.927464+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:27:16.927464+0000","last_undegraded":"2024-05-06T00:27:16.927464+0000","last_fullsized":"2024-05-06T00:27:16.927464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"41'33","reported_seq":148,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932985+0000","last_change":"2024-05-06T00:26:45.733467+0000","last_active":"2024-05-06T00:27:16.932985+0000","last_peered":"2024-05-06T00:27:16.932985+0000","last_clean":"2024-05-06T00:27:16.932985+0000","last_became_active":"2024-05-06T00:26:43.703416+0000","last_became_peered":"2024-05-06T00:26:43.703416+0000","last_unstale":"2024-05-06T00:27:16.932985+0000","last_undegraded":"2024-05-06T00:27:16.932985+0000","last_fullsized":"2024-05-06T00:27:16.932985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:39:19.838424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":32,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":93,"num_write":33,"num_write_kb":396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"41'33","reported_seq":192,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.181160+0000","last_change":"2024-05-06T00:26:45.720211+0000","last_active":"2024-05-06T00:27:17.181160+0000","last_peered":"2024-05-06T00:27:17.181160+0000","last_clean":"2024-05-06T00:27:17.181160+0000","last_became_active":"2024-05-06T00:26:43.705235+0000","last_became_peered":"2024-05-06T00:26:43.705235+0000","last_unstale":"2024-05-06T00:27:17.181160+0000","last_undegraded":"2024-05-06T00:27:17.181160+0000","last_fullsized":"2024-05-06T00:27:17.181160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:50:23.495101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":375845,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":100,"num_read_kb":361,"num_write":32,"num_write_kb":347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932398+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:27:16.932398+0000","last_peered":"2024-05-06T00:27:16.932398+0000","last_clean":"2024-05-06T00:27:16.932398+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:27:16.932398+0000","last_undegraded":"2024-05-06T00:27:16.932398+0000","last_fullsized":"2024-05-06T00:27:16.932398+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"40'145","reported_seq":356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.927117+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:27:16.927117+0000","last_peered":"2024-05-06T00:27:16.927117+0000","last_clean":"2024-05-06T00:27:16.927117+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:27:16.927117+0000","last_undegraded":"2024-05-06T00:27:16.927117+0000","last_fullsized":"2024-05-06T00:27:16.927117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:00:32.019838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":5349480,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":664,"num_read_kb":748,"num_write":211,"num_write_kb":797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920086+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:27:16.920086+0000","last_peered":"2024-05-06T00:27:16.920086+0000","last_clean":"2024-05-06T00:27:16.920086+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:27:16.920086+0000","last_undegraded":"2024-05-06T00:27:16.920086+0000","last_fullsized":"2024-05-06T00:27:16.920086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"41'49","reported_seq":440,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920774+0000","last_change":"2024-05-06T00:26:45.712444+0000","last_active":"2024-05-06T00:27:16.920774+0000","last_peered":"2024-05-06T00:27:16.920774+0000","last_clean":"2024-05-06T00:27:16.920774+0000","last_became_active":"2024-05-06T00:26:43.708328+0000","last_became_peered":"2024-05-06T00:26:43.708328+0000","last_unstale":"2024-05-06T00:27:16.920774+0000","last_undegraded":"2024-05-06T00:27:16.920774+0000","last_fullsized":"2024-05-06T00:27:16.920774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:40.650848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":21321728,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":338,"num_read_kb":2220,"num_write":50,"num_write_kb":778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920786+0000","last_change":"2024-05-06T00:26:35.652664+0000","last_active":"2024-05-06T00:27:16.920786+0000","last_peered":"2024-05-06T00:27:16.920786+0000","last_clean":"2024-05-06T00:27:16.920786+0000","last_became_active":"2024-05-06T00:26:35.652017+0000","last_became_peered":"2024-05-06T00:26:35.652017+0000","last_unstale":"2024-05-06T00:27:16.920786+0000","last_undegraded":"2024-05-06T00:27:16.920786+0000","last_fullsized":"2024-05-06T00:27:16.920786+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:27:34.935271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"42'139","reported_seq":1856,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.339070+0000","last_change":"2024-05-06T00:26:45.725541+0000","last_active":"2024-05-06T00:27:17.339070+0000","last_peered":"2024-05-06T00:27:17.339070+0000","last_clean":"2024-05-06T00:27:17.339070+0000","last_became_active":"2024-05-06T00:26:43.696938+0000","last_became_peered":"2024-05-06T00:26:43.696938+0000","last_unstale":"2024-05-06T00:27:17.339070+0000","last_undegraded":"2024-05-06T00:27:17.339070+0000","last_fullsized":"2024-05-06T00:27:17.339070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:12:11.039209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":58,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2457,"num_read_kb":2220,"num_write":88,"num_write_kb":649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932378+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:27:16.932378+0000","last_peered":"2024-05-06T00:27:16.932378+0000","last_clean":"2024-05-06T00:27:16.932378+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:27:16.932378+0000","last_undegraded":"2024-05-06T00:27:16.932378+0000","last_fullsized":"2024-05-06T00:27:16.932378+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":158,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920821+0000","last_change":"2024-05-06T00:26:30.239639+0000","last_active":"2024-05-06T00:27:16.920821+0000","last_peered":"2024-05-06T00:27:16.920821+0000","last_clean":"2024-05-06T00:27:16.920821+0000","last_became_active":"2024-05-06T00:26:30.237552+0000","last_became_peered":"2024-05-06T00:26:30.237552+0000","last_unstale":"2024-05-06T00:27:16.920821+0000","last_undegraded":"2024-05-06T00:27:16.920821+0000","last_fullsized":"2024-05-06T00:27:16.920821+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:44:57.011856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"42'204","reported_seq":659,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.424149+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:27:17.424149+0000","last_peered":"2024-05-06T00:27:17.424149+0000","last_clean":"2024-05-06T00:27:17.424149+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:27:17.424149+0000","last_undegraded":"2024-05-06T00:27:17.424149+0000","last_fullsized":"2024-05-06T00:27:17.424149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:40:59.218132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":18216088,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":1253,"num_read_kb":1534,"num_write":323,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"41'42","reported_seq":311,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.944800+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:27:16.944800+0000","last_peered":"2024-05-06T00:27:16.944800+0000","last_clean":"2024-05-06T00:27:16.944800+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:27:16.944800+0000","last_undegraded":"2024-05-06T00:27:16.944800+0000","last_fullsized":"2024-05-06T00:27:16.944800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:22:48.068013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":7234048,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":208,"num_read_kb":397,"num_write":40,"num_write_kb":689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"42'197","reported_seq":2191,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.470006+0000","last_change":"2024-05-06T00:26:45.710256+0000","last_active":"2024-05-06T00:27:17.470006+0000","last_peered":"2024-05-06T00:27:17.470006+0000","last_clean":"2024-05-06T00:27:17.470006+0000","last_became_active":"2024-05-06T00:26:43.710572+0000","last_became_peered":"2024-05-06T00:26:43.710572+0000","last_unstale":"2024-05-06T00:27:17.470006+0000","last_undegraded":"2024-05-06T00:27:17.470006+0000","last_fullsized":"2024-05-06T00:27:17.470006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:03:48.514905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":7607843,"num_objects":12,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":3483,"num_read_kb":4838,"num_write":121,"num_write_kb":772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"42'26","reported_seq":165,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.254079+0000","last_change":"2024-05-06T00:26:45.725952+0000","last_active":"2024-05-06T00:27:17.254079+0000","last_peered":"2024-05-06T00:27:17.254079+0000","last_clean":"2024-05-06T00:27:17.254079+0000","last_became_active":"2024-05-06T00:26:43.686980+0000","last_became_peered":"2024-05-06T00:26:43.686980+0000","last_unstale":"2024-05-06T00:27:17.254079+0000","last_undegraded":"2024-05-06T00:27:17.254079+0000","last_fullsized":"2024-05-06T00:27:17.254079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:57:59.257502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":152,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":431,"num_write":25,"num_write_kb":3405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"41'35","reported_seq":392,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.417499+0000","last_change":"2024-05-06T00:26:45.732742+0000","last_active":"2024-05-06T00:27:17.417499+0000","last_peered":"2024-05-06T00:27:17.417499+0000","last_clean":"2024-05-06T00:27:17.417499+0000","last_became_active":"2024-05-06T00:26:43.700115+0000","last_became_peered":"2024-05-06T00:26:43.700115+0000","last_unstale":"2024-05-06T00:27:17.417499+0000","last_undegraded":"2024-05-06T00:27:17.417499+0000","last_fullsized":"2024-05-06T00:27:17.417499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:39:22.920835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":2727936,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":289,"num_read_kb":763,"num_write":35,"num_write_kb":611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"42'23","reported_seq":187,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.418366+0000","last_change":"2024-05-06T00:26:45.711599+0000","last_active":"2024-05-06T00:27:17.418366+0000","last_peered":"2024-05-06T00:27:17.418366+0000","last_clean":"2024-05-06T00:27:17.418366+0000","last_became_active":"2024-05-06T00:26:43.692092+0000","last_became_peered":"2024-05-06T00:26:43.692092+0000","last_unstale":"2024-05-06T00:27:17.418366+0000","last_undegraded":"2024-05-06T00:27:17.418366+0000","last_fullsized":"2024-05-06T00:27:17.418366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:05.056345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":628833,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":105,"num_read_kb":1860,"num_write":21,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"41'52","reported_seq":1121,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.402504+0000","last_change":"2024-05-06T00:26:45.734276+0000","last_active":"2024-05-06T00:27:17.402504+0000","last_peered":"2024-05-06T00:27:17.402504+0000","last_clean":"2024-05-06T00:27:17.402504+0000","last_became_active":"2024-05-06T00:26:43.711491+0000","last_became_peered":"2024-05-06T00:26:43.711491+0000","last_unstale":"2024-05-06T00:27:17.402504+0000","last_undegraded":"2024-05-06T00:27:17.402504+0000","last_fullsized":"2024-05-06T00:27:17.402504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:00:11.880195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":9990144,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1552,"num_read_kb":4490,"num_write":32,"num_write_kb":244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"41'62","reported_seq":174,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.458593+0000","last_change":"2024-05-06T00:26:45.734362+0000","last_active":"2024-05-06T00:27:17.458593+0000","last_peered":"2024-05-06T00:27:17.458593+0000","last_clean":"2024-05-06T00:27:17.458593+0000","last_became_active":"2024-05-06T00:26:43.703972+0000","last_became_peered":"2024-05-06T00:26:43.703972+0000","last_unstale":"2024-05-06T00:27:17.458593+0000","last_undegraded":"2024-05-06T00:27:17.458593+0000","last_fullsized":"2024-05-06T00:27:17.458593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:19:16.880866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":96,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":289,"num_read_kb":258,"num_write":93,"num_write_kb":321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"41'71","reported_seq":831,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.932297+0000","last_change":"2024-05-06T00:26:45.732061+0000","last_active":"2024-05-06T00:27:16.932297+0000","last_peered":"2024-05-06T00:27:16.932297+0000","last_clean":"2024-05-06T00:27:16.932297+0000","last_became_active":"2024-05-06T00:26:43.699883+0000","last_became_peered":"2024-05-06T00:26:43.699883+0000","last_unstale":"2024-05-06T00:27:16.932297+0000","last_undegraded":"2024-05-06T00:27:16.932297+0000","last_fullsized":"2024-05-06T00:27:16.932297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:42:16.152037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":8608768,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":1132,"num_read_kb":2537,"num_write":54,"num_write_kb":436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"41'19","reported_seq":185,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.403953+0000","last_change":"2024-05-06T00:26:45.710994+0000","last_active":"2024-05-06T00:27:17.403953+0000","last_peered":"2024-05-06T00:27:17.403953+0000","last_clean":"2024-05-06T00:27:17.403953+0000","last_became_active":"2024-05-06T00:26:43.710745+0000","last_became_peered":"2024-05-06T00:26:43.710745+0000","last_unstale":"2024-05-06T00:27:17.403953+0000","last_undegraded":"2024-05-06T00:27:17.403953+0000","last_fullsized":"2024-05-06T00:27:17.403953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:38:42.599417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":9511936,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":101,"num_read_kb":2011,"num_write":17,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"41'53","reported_seq":488,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:16.920929+0000","last_change":"2024-05-06T00:26:45.713363+0000","last_active":"2024-05-06T00:27:16.920929+0000","last_peered":"2024-05-06T00:27:16.920929+0000","last_clean":"2024-05-06T00:27:16.920929+0000","last_became_active":"2024-05-06T00:26:43.692971+0000","last_became_peered":"2024-05-06T00:26:43.692971+0000","last_unstale":"2024-05-06T00:27:16.920929+0000","last_undegraded":"2024-05-06T00:27:16.920929+0000","last_fullsized":"2024-05-06T00:27:16.920929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:22:06.588605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":19036672,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":3,"num_read":382,"num_read_kb":754,"num_write":49,"num_write_kb":833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"42'150","reported_seq":3066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-06T00:27:17.317501+0000","last_change":"2024-05-06T00:26:45.711743+0000","last_active":"2024-05-06T00:27:17.317501+0000","last_peered":"2024-05-06T00:27:17.317501+0000","last_clean":"2024-05-06T00:27:17.317501+0000","last_became_active":"2024-05-06T00:26:43.700214+0000","last_became_peered":"2024-05-06T00:26:43.700214+0000","last_unstale":"2024-05-06T00:27:17.317501+0000","last_undegraded":"2024-05-06T00:27:17.317501+0000","last_fullsized":"2024-05-06T00:27:17.317501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:36:59.965708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":11649040,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4402,"num_read_kb":11196,"num_write":100,"num_write_kb":652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":296358079,"num_objects":193,"num_object_clones":47,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":45,"num_read":36616,"num_read_kb":75256,"num_write":3142,"num_write_kb":35468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5529600,"data_stored":5352830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2932,"ondisk_log_size":2932,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640271,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1343488,"data_stored":1080626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640269,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1544192,"data_stored":1196912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640269,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33440,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338400,"statfs":{"total":96636764160,"available":96602521600,"internally_reserved":0,"allocated":2310144,"data_stored":2076056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339640,"statfs":{"total":96636764160,"available":96603791360,"internally_reserved":0,"allocated":1048576,"data_stored":707625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640271,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36456,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335384,"statfs":{"total":96636764160,"available":96599433216,"internally_reserved":0,"allocated":1146880,"data_stored":788703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640269,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33248,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338592,"statfs":{"total":96636764160,"available":96602718208,"internally_reserved":0,"allocated":2109440,"data_stored":1766031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":999826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":607393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":526315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1249280,"data_stored":1304378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1077248,"data_stored":1015602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":876544,"data_stored":899316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:27:20.255 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-05-06T00:27:20.255 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:27:20.255 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-06T00:27:20.255 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2024-05-06T00:27:20.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0 2024-05-06T00:27:20.256 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:27:20.349 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:20.348+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:20.422 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:20.423 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:20.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:20.449 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:20.449+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:20.551 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:20.548+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:20.606 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:20.606 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:20.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:20.662 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:20.660+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:20.753 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:20.753+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:20.784 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:20.784 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:20.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:20.854 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:20.852+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:20.923 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:20.924 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:20.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:20.952 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.3 primary-affinity to 0 (802) 2024-05-06T00:27:20.980 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:20.980+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:21.056 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:21.056+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:21.103 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:21.103 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:21.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:21.160 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:21.160+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:21.258 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:21.256+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:21.283 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:21.284 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:21.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:21.384 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:21.384+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:21.460 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:21.460+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:21.464 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:21.465 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:21.472 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:21.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:21.565 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:21.565+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:21.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:21.625 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:21.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:21.642 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:21.643 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:21.643 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:21.657 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:21.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:21.662 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:21.660+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:21.683 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:21.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:21.763 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:21.761+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:21.777 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:21.777 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:21.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:21.881 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:21.881+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:21.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:21.942 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:21.967 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:21.965+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.172227669000002, 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:27:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:27:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.172211466, 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:27:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2527 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch ping cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.816117516999999, 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16552214100000001, 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2527 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160471+0000", 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.206e-06 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160477+0000", 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7440000000000001e-06 2024-05-06T00:27:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160491+0000", 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3739e-05 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160506+0000", 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.473e-05 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160712+0000", 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020643899999999999 2024-05-06T00:27:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160736+0000", 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3676000000000002e-05 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325517+0000", 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16478073700000001 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325685+0000", 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016879300000000001 2024-05-06T00:27:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325992+0000", 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030607699999999998 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2558 3.1f 3:fcc12fb6:::rbd_data.110cc8beeb04.0000000000000000:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.478195420999999, 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067308660000000003, 2024-05-06T00:27:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2558 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498392+0000", 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498394+0000", 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2469999999999998e-06 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498404+0000", 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0577000000000001e-05 2024-05-06T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498435+0000", 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0743e-05 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498452+0000", 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6650000000000002e-05 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498620+0000", 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168202 2024-05-06T00:27:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498840+0000", 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022057199999999999 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.499360+0000", 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000519149 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502603+0000", 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032430419999999998 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504911+0000", 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023087350000000001 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505025+0000", 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011343 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505122+0000", 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7518999999999999e-05 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1858 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.475695333000001, 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069279169999999996, 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1858 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500892+0000", 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500894+0000", 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.587e-06 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500907+0000", 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.29e-05 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500940+0000", 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2709999999999997e-05 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.500961+0000", 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0916000000000001e-05 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507119+0000", 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061583690000000003 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507405+0000", 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028573800000000001 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507820+0000", 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000414697 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2571 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.475317878999999, 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079485920000000009, 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2571 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501269+0000", 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501271+0000", 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3700000000000002e-06 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501290+0000", 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8505999999999999e-05 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501321+0000", 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.099e-05 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501340+0000", 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8987e-05 2024-05-06T00:27:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508595+0000", 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072554910000000002 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508787+0000", 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019116399999999999 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509218+0000", 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043108399999999998 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1859 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.474506239, 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078054739999999997, 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1859 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502081+0000", 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502083+0000", 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7070000000000001e-06 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502097+0000", 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3447e-05 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502130+0000", 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3300000000000003e-05 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502151+0000", 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0665999999999998e-05 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509314+0000", 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071635980000000002 2024-05-06T00:27:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509492+0000", 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000177975 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509886+0000", 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000393781 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2572 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.474427675000001, 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085800230000000009, 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2572 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502159+0000", 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502162+0000", 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.993e-06 2024-05-06T00:27:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502177+0000", 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.506e-05 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502206+0000", 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8379000000000001e-05 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502226+0000", 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9927000000000001e-05 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510022+0000", 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077960959999999998 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510238+0000", 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021639200000000001 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510739+0000", 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00050117599999999999 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2573 3.12 3:4926f074:::rbd_data.110cc8beeb04.0000000000000006:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.473849247, 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066001020000000001, 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2573 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502738+0000", 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502740+0000", 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.88e-06 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502749+0000", 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8400000000000007e-06 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502771+0000", 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1622999999999999e-05 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502788+0000", 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7e-05 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502947+0000", 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015869300000000001 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503160+0000", 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000213291 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503643+0000", 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048288900000000002 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507989+0000", 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043459170000000004 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509041+0000", 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010519209999999999 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509218+0000", 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017686899999999999 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509338+0000", 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120179 2024-05-06T00:27:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2580 3.12 3:4e19e050:::rbd_data.110cc8beeb04.000000000000002d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.473734294, 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079317220000000004, 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2580 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502853+0000", 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502854+0000", 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4360000000000001e-06 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502862+0000", 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9140000000000008e-06 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502882+0000", 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9885999999999999e-05 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502896+0000", 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4219999999999999e-05 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505092+0000", 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021958530000000002 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505289+0000", 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019717899999999999 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505714+0000", 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042437900000000001 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508626+0000", 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029123479999999999 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510454+0000", 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001828292 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510664+0000", 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000209829 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.510784+0000", 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120386 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2581 3.1b 3:d89117cf:::rbd_data.110cc8beeb04.0000000000000033:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.473663910999999, 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010296013, 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2581 2024-05-06T00:27:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502923+0000", 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502924+0000", 2024-05-06T00:27:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.243e-06 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502931+0000", 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502948+0000", 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7374e-05 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502962+0000", 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3653e-05 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503093+0000", 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013094499999999999 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503278+0000", 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000185582 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503709+0000", 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043043999999999998 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507476+0000", 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003766901 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513089+0000", 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005613743 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513164+0000", 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4276999999999994e-05 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513219+0000", 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5432000000000001e-05 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1860 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.472118147, 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096507989999999998, 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1860 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504469+0000", 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504472+0000", 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.027e-06 2024-05-06T00:27:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504488+0000", 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6629999999999998e-05 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504523+0000", 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4178999999999999e-05 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.504544+0000", 2024-05-06T00:27:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1627000000000001e-05 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513785+0000", 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092403420000000003 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.513895+0000", 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110796 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.514120+0000", 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000224198 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2586 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051770034032] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.471265089999999, 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070106719999999999, 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2586 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505322+0000", 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505323+0000", 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.243e-06 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505330+0000", 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1169999999999998e-06 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505349+0000", 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8482999999999999e-05 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505368+0000", 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9576999999999999e-05 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511761+0000", 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063923230000000001 2024-05-06T00:27:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.511904+0000", 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014312600000000001 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.512333+0000", 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042880300000000001 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2591 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [notify cookie 140051769985984] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.467616102000001, 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066352190000000004, 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2591 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508971+0000", 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508973+0000", 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.39e-06 2024-05-06T00:27:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.508983+0000", 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0254e-05 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509010+0000", 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6398999999999999e-05 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509033+0000", 2024-05-06T00:27:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.298e-05 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515092+0000", 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060590740000000002 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515201+0000", 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108602 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.515606+0000", 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040551999999999998 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4737.0:869 3.1f 3:fd5b52f4:::rbd_data.110c6af4f54c.0000000000000000:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:10.026885+0000", 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.949701831, 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079024339999999998, 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4737", 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 869 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026885+0000", 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026885+0000", 2024-05-06T00:27:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026887+0000", 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1670000000000002e-06 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026903+0000", 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5288999999999999e-05 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026924+0000", 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1518999999999998e-05 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.026934+0000", 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5219999999999996e-06 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027035+0000", 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010157000000000001 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027170+0000", 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134843 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027437+0000", 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026655400000000001 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.031220+0000", 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037831499999999999 2024-05-06T00:27:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.034669+0000", 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003449021 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.034719+0000", 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0659999999999999e-05 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.034788+0000", 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8139000000000006e-05 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4737.0:875 3.1b 3:dd6c69eb:::rbd_data.110c6af4f54c.0000000000000004:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:10.027326+0000", 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.949261064, 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010240862999999999, 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4737", 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 875 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027326+0000", 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027326+0000", 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027327+0000", 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4139999999999999e-06 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027334+0000", 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6889999999999998e-06 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027352+0000", 2024-05-06T00:27:22.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8168e-05 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027370+0000", 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8079000000000001e-05 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027449+0000", 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8417000000000005e-05 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027546+0000", 2024-05-06T00:27:22.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7602999999999994e-05 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027793+0000", 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024628399999999999 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028868+0000", 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010758300000000001 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037454+0000", 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085859209999999998 2024-05-06T00:27:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037516+0000", 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1569000000000006e-05 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037567+0000", 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0889e-05 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4737.0:878 3.1 3:832629b4:::rbd_data.110c6af4f54c.0000000000000005:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:10.027395+0000", 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.949191552, 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010172182, 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4737", 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 878 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027395+0000", 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027395+0000", 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027396+0000", 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027401+0000", 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6539999999999998e-06 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027411+0000", 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0513000000000001e-05 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027418+0000", 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2250000000000003e-06 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027483+0000", 2024-05-06T00:27:22.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4262000000000002e-05 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027561+0000", 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8595000000000002e-05 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.027749+0000", 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018780899999999999 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028942+0000", 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001192654 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037428+0000", 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084866060000000007 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037489+0000", 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0686000000000001e-05 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037568+0000", 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8559000000000005e-05 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4737.0:889 3.6 3:645a233e:::rbd_data.110c6af4f54c.000000000000000b:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.948404907, 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093640990000000007, 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4737", 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/964576699", 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 889 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028182+0000", 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028183+0000", 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0210000000000001e-06 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028188+0000", 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2870000000000004e-06 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028199+0000", 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0861e-05 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028207+0000", 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2199999999999992e-06 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028285+0000", 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7979999999999995e-05 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028387+0000", 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010209600000000001 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.028589+0000", 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020201299999999999 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.029814+0000", 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012242500000000001 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037413+0000", 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075991619999999996 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037482+0000", 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8792000000000001e-05 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:10.037546+0000", 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4417000000000003e-05 2024-05-06T00:27:22.073 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:4152 3.1b 3:d9660d20:::rbd_data.110cf57807c6.000000000000009f:head [set-alloc-hint object_size 1048576 write_size 1048576,write 499712~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.943895+0000", 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.0326919910000001, 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006849852, 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.074 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4152 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.943895+0000", 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.943895+0000", 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.943897+0000", 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1730000000000002e-06 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.943910+0000", 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2872e-05 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.943928+0000", 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.804e-05 2024-05-06T00:27:22.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.943938+0000", 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0037000000000001e-05 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.944047+0000", 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109323 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:27:22.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.944057+0000", 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6539999999999994e-06 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.945758+0000", 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017009499999999999 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.945854+0000", 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5883000000000003e-05 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.946002+0000", 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807500000000001 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.946752+0000", 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00074986400000000004 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.950638+0000", 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038860980000000002 2024-05-06T00:27:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.950700+0000", 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1608999999999999e-05 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.950745+0000", 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5274000000000001e-05 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:4226 3.1b 3:d8840528:::rbd_data.110cf57807c6.0000000000000025:head [write 442368~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.976483+0000", 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.0001041609999999, 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086794980000000008, 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4226 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.976483+0000", 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.976483+0000", 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.976484+0000", 2024-05-06T00:27:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3399999999999997e-07 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.976493+0000", 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1130000000000007e-06 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.976503+0000", 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0052e-05 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.976511+0000", 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0330000000000008e-06 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.977802+0000", 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012911680000000001 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.977881+0000", 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9291999999999999e-05 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.978011+0000", 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129799 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.978634+0000", 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00062253299999999998 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.985059+0000", 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064256770000000003 2024-05-06T00:27:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.985118+0000", 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8202000000000003e-05 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.985162+0000", 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4694999999999998e-05 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.087 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:22.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:22.089 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:22.085+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:22.110 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone10 from 0xafef800 (overlap 0x3cec00) to 0x61600 2024-05-06T00:27:22.188 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:22.185+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:22.257 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:27:22.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:22.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:22.261 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:22.261 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:22.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:22.290 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:22.288+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:22.390 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:22.389+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:22.470 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:22.470 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:22.470 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:22.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:22.491 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:22.488+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:22.593 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:22.589+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:22.631 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-05-06T00:27:22.661 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:228 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 397312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.063150851, 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072743440000000003, 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:27:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 228 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597015+0000", 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597016+0000", 2024-05-06T00:27:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.443e-06 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597023+0000", 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8340000000000001e-06 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597044+0000", 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0876000000000001e-05 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597055+0000", 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1314e-05 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597176+0000", 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120565 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597192+0000", 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6337e-05 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603743+0000", 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065506820000000004 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.603976+0000", 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000233355 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604289+0000", 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031293800000000002 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4420.0:229 3.19 3:994dc6a3:::rbd_data.110cdb7de3c5.0000000000000005:head [sparse-read 528384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.062992951999998, 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075584060000000002, 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4420", 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3269360150", 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 229 2024-05-06T00:27:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597173+0000", 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597175+0000", 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8929999999999999e-06 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597184+0000", 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2040000000000003e-06 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597200+0000", 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5642999999999999e-05 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597209+0000", 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4329999999999993e-06 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597301+0000", 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1588999999999993e-05 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.597309+0000", 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604360+0000", 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070513030000000001 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604516+0000", 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015564999999999999 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:51.604731+0000", 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000215267 2024-05-06T00:27:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:165 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.976009210000001, 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010266747, 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 165 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684157+0000", 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684158+0000", 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6929999999999999e-06 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684170+0000", 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1933000000000001e-05 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684198+0000", 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8266999999999999e-05 2024-05-06T00:27:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.684221+0000", 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2736999999999999e-05 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687824+0000", 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003602863 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.687915+0000", 2024-05-06T00:27:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0804999999999995e-05 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.688264+0000", 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034917100000000001 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.693979+0000", 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057148959999999997 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694148+0000", 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168951 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694315+0000", 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000167016 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.694423+0000", 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108415 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:298 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.900674558999999, 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089449070000000002, 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 298 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759491+0000", 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759493+0000", 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8029999999999999e-06 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759503+0000", 2024-05-06T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4769999999999992e-06 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759525+0000", 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2067999999999999e-05 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.759540+0000", 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5296999999999999e-05 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.762625+0000", 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030855399999999999 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.762723+0000", 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7750000000000004e-05 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.763171+0000", 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044755199999999999 2024-05-06T00:27:22.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766326+0000", 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031554080000000002 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768328+0000", 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020015530000000001 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768387+0000", 2024-05-06T00:27:22.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9283000000000001e-05 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.768436+0000", 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9175999999999998e-05 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:300 3.4 3:26d09f8b:::rbd_object_map.110cc8beeb04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:22.684 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.893752256999999, 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077869669999999997, 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 300 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766414+0000", 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766415+0000", 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.536e-06 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766423+0000", 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2849999999999995e-06 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766435+0000", 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2121e-05 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766443+0000", 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9140000000000008e-06 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766490+0000", 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6517999999999997e-05 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.766495+0000", 2024-05-06T00:27:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769158+0000", 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026631390000000001 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769268+0000", 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010981599999999999 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.769603+0000", 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033493500000000002 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.773467+0000", 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038639920000000001 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774044+0000", 2024-05-06T00:27:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00057753000000000004 2024-05-06T00:27:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774139+0000", 2024-05-06T00:27:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5315999999999993e-05 2024-05-06T00:27:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.774201+0000", 2024-05-06T00:27:22.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1054000000000002e-05 2024-05-06T00:27:22.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.694 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:324 3.19 3:99a6ca97:::rbd_data.110cc8beeb04.0000000000000013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.869156822000001, 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062413850000000003, 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 324 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791009+0000", 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791010+0000", 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0449999999999999e-06 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791015+0000", 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4689999999999996e-06 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791030+0000", 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4892000000000001e-05 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791038+0000", 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8460000000000008e-06 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791106+0000", 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7994e-05 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791190+0000", 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3337000000000005e-05 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791293+0000", 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103757 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795405+0000", 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041115479999999996 2024-05-06T00:27:22.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797183+0000", 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001778097 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797213+0000", 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.004e-05 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797250+0000", 2024-05-06T00:27:22.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7360000000000001e-05 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:328 3.19 3:9fbbd60a:::rbd_data.110cc8beeb04.000000000000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.86900851, 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063090830000000001, 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 328 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791157+0000", 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791158+0000", 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7000000000000003e-07 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791165+0000", 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3220000000000004e-06 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791178+0000", 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2078e-05 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791184+0000", 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791609+0000", 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042469600000000001 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.791782+0000", 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017316400000000001 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.792020+0000", 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023781599999999999 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.795867+0000", 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038475010000000001 2024-05-06T00:27:22.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797389+0000", 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015214339999999999 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797418+0000", 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8977999999999999e-05 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:55.797466+0000", 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8489999999999998e-05 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1015 3.8 e29/18 3:1391aa96:::rbd_data.110cc8beeb04.0000000000000025:head v 29'6, mlcod=29'4)", 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.406103405, 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075649460000000003, 2024-05-06T00:27:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254062+0000", 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254063+0000", 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4099999999999997e-07 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254072+0000", 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6370000000000005e-06 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254084+0000", 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1725999999999999e-05 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254091+0000", 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.379e-06 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254155+0000", 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3906999999999996e-05 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254177+0000", 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1526e-05 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261558+0000", 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073814409999999999 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261627+0000", 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9388999999999996e-05 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1017 3.7 e29/18 3:e219a59f:::rbd_data.110cc8beeb04.0000000000000027:head v 29'11, mlcod=29'9)", 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.406058417000001, 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075152739999999997, 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254107+0000", 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254108+0000", 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254112+0000", 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254119+0000", 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7800000000000003e-06 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254124+0000", 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8509999999999998e-06 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254189+0000", 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4757000000000004e-05 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254209+0000", 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0307000000000001e-05 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261566+0000", 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007356596 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261623+0000", 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7111000000000003e-05 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1022 3.8 e29/18 3:17fe3906:::rbd_data.110cc8beeb04.0000000000000036:head v 29'7, mlcod=29'4)", 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.405896300999999, 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073889450000000001, 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254269+0000", 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254270+0000", 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254275+0000", 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6120000000000001e-06 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254283+0000", 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2199999999999992e-06 2024-05-06T00:27:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254289+0000", 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9880000000000001e-06 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254360+0000", 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1062000000000005e-05 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254379+0000", 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9194e-05 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261632+0000", 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00725255 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261658+0000", 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6857000000000001e-05 2024-05-06T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4578.0:1018 3.7 e29/18 3:e3d3a9cb:::rbd_data.110cc8beeb04.0000000000000028:head v 29'12, mlcod=29'9)", 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.405844473999998, 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073213799999999997, 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254321+0000", 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254322+0000", 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5400000000000002e-07 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254326+0000", 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5979999999999999e-06 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254334+0000", 2024-05-06T00:27:22.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1489999999999997e-06 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254339+0000", 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1399999999999999e-06 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254387+0000", 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8102000000000001e-05 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.254403+0000", 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5982e-05 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261626+0000", 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072235140000000003 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:56.261643+0000", 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6441000000000001e-05 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2574 3.4 3:218edb7b:::rbd_data.110cc8beeb04.0000000000000007:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.157329057999998, 2024-05-06T00:27:22.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006796074, 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2574 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:22.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502837+0000", 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502839+0000", 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7329999999999999e-06 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502852+0000", 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2203e-05 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502884+0000", 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2543000000000003e-05 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.502906+0000", 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2027000000000001e-05 2024-05-06T00:27:22.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503066+0000", 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159946 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503260+0000", 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019375100000000001 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.503688+0000", 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042773399999999998 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.506368+0000", 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002680689 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509377+0000", 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003008667 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509527+0000", 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149498 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.509633+0000", 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1279 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:22.733 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.999051818000002, 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084436479999999998, 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1279 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661114+0000", 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661117+0000", 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6929999999999999e-06 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661132+0000", 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5153999999999999e-05 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661170+0000", 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7886000000000003e-05 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661188+0000", 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7816e-05 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661295+0000", 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001072 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.661597+0000", 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030253000000000002 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.662320+0000", 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072273200000000002 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669339+0000", 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070190219999999998 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669377+0000", 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7935999999999997e-05 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669495+0000", 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117599 2024-05-06T00:27:22.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669558+0000", 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3079999999999999e-05 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1293 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.996744259, 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066635549999999998, 2024-05-06T00:27:22.739 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1293 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663422+0000", 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663424+0000", 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6599999999999999e-06 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663439+0000", 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4633e-05 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663471+0000", 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1986999999999997e-05 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663491+0000", 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0435999999999999e-05 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663643+0000", 2024-05-06T00:27:22.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151929 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663863+0000", 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021996200000000001 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.664839+0000", 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00097599500000000001 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669583+0000", 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047435070000000001 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669694+0000", 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011181899999999999 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669963+0000", 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026898800000000001 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670085+0000", 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012163900000000001 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1294 3.0 3:01e27c79:::rbd_object_map.110c7776112a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.996444335, 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086615830000000005, 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1294 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663721+0000", 2024-05-06T00:27:22.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663724+0000", 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.61e-06 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663739+0000", 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.447e-05 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663768+0000", 2024-05-06T00:27:22.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9957000000000001e-05 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.663789+0000", 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0420000000000001e-05 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.667197+0000", 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003407963 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.667372+0000", 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000175532 2024-05-06T00:27:22.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.668276+0000", 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00090397299999999997 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670141+0000", 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001864239 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672093+0000", 2024-05-06T00:27:22.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019525650000000001 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672271+0000", 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017802900000000001 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672383+0000", 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011182499999999999 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.750 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4528.0:1303 3.4 3:214da2ae:::rbd_data.110c7776112a.0000000000000007:head [set-alloc-hint object_size 33554432 write_size 33554432,write 2125824~23552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.990174104000001, 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063999010000000004, 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4528", 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3313871951", 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1303 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669992+0000", 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.669994+0000", 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8169999999999999e-06 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670033+0000", 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9113000000000003e-05 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670057+0000", 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4334000000000001e-05 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.670078+0000", 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1539000000000002e-05 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672106+0000", 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00202797 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672301+0000", 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019495800000000001 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.672680+0000", 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037862600000000002 2024-05-06T00:27:22.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.675864+0000", 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003183745 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676177+0000", 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031343499999999998 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676313+0000", 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013609700000000001 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:03.676392+0000", 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8267000000000001e-05 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:3509 3.b 3:d0d4a4f7:::rbd_data.110cf57807c6.0000000000000014:head [write 794624~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.722411+0000", 2024-05-06T00:27:22.756 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9377552119999999, 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064962279999999997, 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 3509 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.722411+0000", 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.722411+0000", 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.722413+0000", 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4669999999999998e-06 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.722435+0000", 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2337e-05 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.722462+0000", 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6914e-05 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.722482+0000", 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9967e-05 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.722612+0000", 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012950400000000001 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.722782+0000", 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016993299999999999 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.723072+0000", 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028985399999999998 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.724810+0000", 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017387279999999999 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.728769+0000", 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039588150000000001 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.728850+0000", 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0817999999999998e-05 2024-05-06T00:27:22.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.728907+0000", 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6891000000000001e-05 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:4144 3.4 3:2435e2ec:::rbd_data.110cf57807c6.0000000000000098:head [set-alloc-hint object_size 1048576 write_size 1048576,write 503808~4608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.933988+0000", 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7261782299999999, 2024-05-06T00:27:22.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064200139999999999, 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4144 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.933988+0000", 2024-05-06T00:27:22.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.933988+0000", 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.933988+0000", 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2600000000000001e-07 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.934002+0000", 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3478999999999999e-05 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.934014+0000", 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2493e-05 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.934026+0000", 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1286e-05 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.934089+0000", 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3517999999999997e-05 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.934200+0000", 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000111162 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.934441+0000", 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000240225 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.936059+0000", 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016179149999999999 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.940300+0000", 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042412049999999996 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.940360+0000", 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9824000000000002e-05 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.940408+0000", 2024-05-06T00:27:22.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7981e-05 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6895 3.4 3:21bb1743:::rbd_data.110cf57807c6.000000000000000f:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.122905+0000", 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.53726122300000001, 2024-05-06T00:27:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088869259999999999, 2024-05-06T00:27:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6895 2024-05-06T00:27:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.122905+0000", 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.122905+0000", 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.122913+0000", 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4360000000000002e-06 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.122922+0000", 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8599999999999999e-06 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.122945+0000", 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2654e-05 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.122967+0000", 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.264e-05 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.125670+0000", 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027029279999999998 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.125894+0000", 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022363299999999999 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.126303+0000", 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040889300000000002 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.128528+0000", 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022257140000000002 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.131460+0000", 2024-05-06T00:27:22.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002931314 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.131682+0000", 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000222489 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.131791+0000", 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010936499999999999 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6905 3.b 3:d0d4a4f7:::rbd_data.110cf57807c6.0000000000000014:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.128713+0000", 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.53145294899999995, 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078734740000000001, 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6905 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.128713+0000", 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.128713+0000", 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.128715+0000", 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5399999999999998e-06 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.128728+0000", 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2503e-05 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.128762+0000", 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4220000000000001e-05 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.128777+0000", 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4864e-05 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.128928+0000", 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000150741 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.129138+0000", 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000210099 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.129557+0000", 2024-05-06T00:27:22.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041953699999999999 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.131660+0000", 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021027229999999999 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.136459+0000", 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047986679999999999 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.136530+0000", 2024-05-06T00:27:22.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1581999999999993e-05 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.136586+0000", 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5996999999999999e-05 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:22.781 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:22.783 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:22.783 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:22.780+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:22.883 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:22.880+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:22.967 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:27:22.969 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:22.969 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:22.969 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:22.969 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:22.984 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:22.981+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:22.989 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:23.085 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:23.085+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:23.161 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:23.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:23.162 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:23.162 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:23.162 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:23.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:23.217 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:23.217+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:23.287 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:23.285+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:23.316 INFO:teuthology.orchestra.run.smithi186.stdout:1685 read 0x8138000 thru 0x813a0d8 (0x20d9 bytes) 2024-05-06T00:27:23.317 INFO:teuthology.orchestra.run.smithi186.stdout:1686 punch from 0x48d1e00 to 0x48da000, (0x8200 bytes) 2024-05-06T00:27:23.334 INFO:teuthology.orchestra.run.smithi186.stdout:1689 punch from 0x556c600 to 0x557ae00, (0xe800 bytes) 2024-05-06T00:27:23.342 INFO:teuthology.orchestra.run.smithi186.stdout:1690 write 0x7eb7c00 thru 0x7ebb867 (0x3c68 bytes) 2024-05-06T00:27:23.343 INFO:teuthology.orchestra.run.smithi186.stdout:1691 clone 12 order 23 su 131072 sc 9 2024-05-06T00:27:23.344 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:23.344 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:23.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:23.344 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:23.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.344 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2567 3.1e 3:7ff04bbe:::rbd_data.110cc8beeb04.0000000000000005:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.842640959000001, 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091482830000000001, 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2567 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498657+0000", 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.75e-06 2024-05-06T00:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498666+0000", 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3030000000000005e-06 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498690+0000", 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-05 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498707+0000", 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7326999999999999e-05 2024-05-06T00:27:23.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501155+0000", 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002447884 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501468+0000", 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031281100000000002 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501983+0000", 2024-05-06T00:27:23.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051484999999999996 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505388+0000", 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034055240000000001 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507550+0000", 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021622690000000001 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507694+0000", 2024-05-06T00:27:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014377899999999999 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507804+0000", 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109486 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1779 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:23.350 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.685180913, 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095752939999999998, 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1779 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656116+0000", 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9699999999999995e-07 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656120+0000", 2024-05-06T00:27:23.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5939999999999998e-06 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656128+0000", 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0309999999999998e-06 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656133+0000", 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.659748+0000", 2024-05-06T00:27:23.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003615231 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.659987+0000", 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023881900000000001 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.660658+0000", 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067093800000000005 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.663069+0000", 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024110109999999998 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665388+0000", 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023186579999999999 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665608+0000", 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022027 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665691+0000", 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2698e-05 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1841 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.655843332, 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011157446999999999, 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1841 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:23.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685454+0000", 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2200000000000002e-07 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685459+0000", 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4820000000000002e-06 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685469+0000", 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8020000000000003e-06 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685476+0000", 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-06 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.689403+0000", 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003927346 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.689582+0000", 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017826300000000001 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.690411+0000", 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00082928899999999998 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696488+0000", 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060775960000000002 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696529+0000", 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0416000000000001e-05 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696571+0000", 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1931000000000002e-05 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696610+0000", 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9660000000000003e-05 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.361 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1848 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.654358025000001, 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011575561999999999, 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1848 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686939+0000", 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.136e-06 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686946+0000", 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2790000000000004e-06 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686959+0000", 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3774000000000001e-05 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686969+0000", 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2070000000000001e-06 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.691625+0000", 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046563450000000001 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.691714+0000", 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8651000000000002e-05 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692727+0000", 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010135400000000001 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696654+0000", 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039264529999999999 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698290+0000", 2024-05-06T00:27:23.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016358880000000001 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698414+0000", 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000124082 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698514+0000", 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100207 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1849 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.654143899999999, 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013152664, 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1849 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687153+0000", 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6599999999999994e-07 2024-05-06T00:27:23.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687172+0000", 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8868999999999998e-05 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687187+0000", 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4491999999999999e-05 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687196+0000", 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6039999999999999e-06 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.694179+0000", 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069822119999999998 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.694396+0000", 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000217856 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.695435+0000", 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001038349 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700028+0000", 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045936090000000002 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700067+0000", 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9048999999999998e-05 2024-05-06T00:27:23.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700203+0000", 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013551700000000001 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700305+0000", 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102141 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.373 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1850 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.653513737000001, 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01443141, 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1850 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687784+0000", 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.311e-06 2024-05-06T00:27:23.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687789+0000", 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5289999999999999e-06 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687809+0000", 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9911e-05 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687818+0000", 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9549999999999998e-06 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696755+0000", 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089369069999999991 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696977+0000", 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000221337 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.697535+0000", 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00055843500000000003 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700358+0000", 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028226850000000001 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702017+0000", 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001659537 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702117+0000", 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9984999999999999e-05 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702214+0000", 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6817999999999994e-05 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.379 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1869 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.649088151000001, 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011357313000000001, 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1869 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692210+0000", 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.514e-06 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692219+0000", 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2399999999999996e-06 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692240+0000", 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1537000000000001e-05 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692257+0000", 2024-05-06T00:27:23.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6670000000000001e-05 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698589+0000", 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063314920000000002 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698673+0000", 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4358999999999996e-05 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.699065+0000", 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039182699999999997 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702254+0000", 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031890780000000001 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703473+0000", 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012188730000000001 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703526+0000", 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3588999999999998e-05 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703565+0000", 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9134000000000002e-05 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1870 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:23.385 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.648984146, 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011641416999999999, 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1870 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692314+0000", 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.42e-06 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692323+0000", 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1409999999999994e-06 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692346+0000", 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2955999999999998e-05 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692363+0000", 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6923999999999999e-05 2024-05-06T00:27:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700432+0000", 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080692569999999998 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700598+0000", 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166422 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.701369+0000", 2024-05-06T00:27:23.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00077104099999999998 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703579+0000", 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022098 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703712+0000", 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133378 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703863+0000", 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015091700000000001 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703954+0000", 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0161000000000006e-05 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2082 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:23.391 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.527254879000001, 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098291899999999998, 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2082 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814043+0000", 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2720000000000001e-06 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814049+0000", 2024-05-06T00:27:23.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814061+0000", 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1815e-05 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814069+0000", 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4630000000000001e-06 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.816625+0000", 2024-05-06T00:27:23.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025561239999999999 2024-05-06T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.816767+0000", 2024-05-06T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014136199999999999 2024-05-06T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.817683+0000", 2024-05-06T00:27:23.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091655999999999997 2024-05-06T00:27:23.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823412+0000", 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057287509999999998 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823578+0000", 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016635 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823763+0000", 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184172 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823871+0000", 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108068 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2083 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.527160243000001, 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099999149999999998, 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2083 2024-05-06T00:27:23.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.399 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814137+0000", 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9299999999999997e-07 2024-05-06T00:27:23.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814141+0000", 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5750000000000002e-06 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814151+0000", 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4450000000000002e-06 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814160+0000", 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3279999999999995e-06 2024-05-06T00:27:23.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.818686+0000", 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045255809999999999 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.818863+0000", 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017779199999999999 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.819847+0000", 2024-05-06T00:27:23.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098399199999999994 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823546+0000", 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036989969999999999 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823898+0000", 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035146899999999998 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824030+0000", 2024-05-06T00:27:23.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000132392 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824136+0000", 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105651 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2084 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:23.406 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.527072849, 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011218071, 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2084 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814225+0000", 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.13e-06 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814230+0000", 2024-05-06T00:27:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1150000000000002e-06 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814243+0000", 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3101999999999999e-05 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814250+0000", 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8759999999999999e-06 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.821215+0000", 2024-05-06T00:27:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069655439999999997 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.821467+0000", 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025156400000000002 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.822255+0000", 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00078842500000000004 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824161+0000", 2024-05-06T00:27:23.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019054429999999999 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825286+0000", 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001125687 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825375+0000", 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9013999999999999e-05 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825442+0000", 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6171000000000001e-05 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4646.0:109 3.8 3:11ca53b4:::rbd_object_map.110cc6eb3028:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7839582529999998, 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010787722, 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.412 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4646", 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/4090392681", 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 109 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557341+0000", 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.027e-06 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557357+0000", 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5926999999999998e-05 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557392+0000", 2024-05-06T00:27:23.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5264e-05 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557414+0000", 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1557e-05 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.561705+0000", 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042916580000000003 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.561768+0000", 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2188999999999997e-05 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.562103+0000", 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033499199999999999 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.567706+0000", 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056037730000000003 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568002+0000", 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029544 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568076+0000", 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3796999999999998e-05 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568126+0000", 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0098e-05 2024-05-06T00:27:23.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4863.0:3614 3.d e43/18 3:b17d50d3:::rbd_object_map.110cf57807c6:head v 43'210, mlcod=43'205)", 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.765118+0000", 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5761779029999996, 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0087561240000000005, 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765118+0000", 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765118+0000", 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765126+0000", 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-06 2024-05-06T00:27:23.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765144+0000", 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8331000000000001e-05 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765171+0000", 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6420000000000001e-05 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.765189+0000", 2024-05-06T00:27:23.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7827000000000001e-05 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.766291+0000", 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001102223 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.766346+0000", 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5050999999999999e-05 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.773757+0000", 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074109479999999997 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.773875+0000", 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117564 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4863.0:3623 3.d e43/18 3:b17d50d3:::rbd_object_map.110cf57807c6:head v 43'218, mlcod=43'208)", 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:23.422 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5668610420000002, 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097994039999999994, 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:23.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774437+0000", 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774450+0000", 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3257e-05 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774476+0000", 2024-05-06T00:27:23.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6108000000000001e-05 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774496+0000", 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9843e-05 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.777187+0000", 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026909 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.777241+0000", 2024-05-06T00:27:23.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4104000000000002e-05 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784162+0000", 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069210519999999996 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784235+0000", 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2596999999999996e-05 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4863.0:3625 3.d e43/18 3:b17d50d3:::rbd_object_map.110cf57807c6:head v 43'219, mlcod=43'208)", 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.775265+0000", 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5660313420000005, 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090135049999999998, 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775265+0000", 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775265+0000", 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775267+0000", 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.584e-06 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775284+0000", 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7283999999999999e-05 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775315+0000", 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1192999999999998e-05 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.775334+0000", 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8507000000000001e-05 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.778537+0000", 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032031659999999999 2024-05-06T00:27:23.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.778617+0000", 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0464999999999996e-05 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784238+0000", 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056206149999999998 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784278+0000", 2024-05-06T00:27:23.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0691000000000001e-05 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6973 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152046+0000", 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.1892501449999999, 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090033119999999994, 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:23.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6973 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152046+0000", 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152046+0000", 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152047+0000", 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4000000000000001e-07 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152051+0000", 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7730000000000001e-06 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152067+0000", 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6497000000000001e-05 2024-05-06T00:27:23.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152081+0000", 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3947e-05 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.156281+0000", 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041999819999999997 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.156436+0000", 2024-05-06T00:27:23.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015487399999999999 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.156951+0000", 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051482100000000005 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.160214+0000", 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003263691 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.160976+0000", 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076118699999999997 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.161018+0000", 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2466999999999998e-05 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.161049+0000", 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1433000000000002e-05 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6974 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152092+0000", 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.1892038249999999, 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010723718, 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6974 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.437 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152092+0000", 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152092+0000", 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152093+0000", 2024-05-06T00:27:23.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4700000000000001e-07 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152097+0000", 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.216e-06 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152107+0000", 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2469999999999997e-06 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152114+0000", 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3300000000000001e-06 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.157905+0000", 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005791175 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.157965+0000", 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9505e-05 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:23.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.158220+0000", 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025549500000000001 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162593+0000", 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043727949999999996 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162689+0000", 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6514000000000005e-05 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162753+0000", 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4015000000000003e-05 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162816+0000", 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2779000000000003e-05 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6975 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:23.442 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152139+0000", 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.189157061, 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011796399000000001, 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6975 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152139+0000", 2024-05-06T00:27:23.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152139+0000", 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152140+0000", 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9000000000000005e-07 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152147+0000", 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1729999999999998e-06 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152167+0000", 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9734000000000001e-05 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152184+0000", 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6387000000000001e-05 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.158668+0000", 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064840829999999999 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.158728+0000", 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0868000000000001e-05 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.159005+0000", 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000276879 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162611+0000", 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003606106 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.163838+0000", 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012269819999999999 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.163881+0000", 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2253999999999999e-05 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.163936+0000", 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4942999999999999e-05 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6976 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152202+0000", 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.1890946170000001, 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012116782, 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.448 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6976 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152202+0000", 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152202+0000", 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152203+0000", 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8000000000000004e-07 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152210+0000", 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2259999999999999e-06 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152226+0000", 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6470999999999999e-05 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152240+0000", 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3665999999999999e-05 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.159447+0000", 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072069040000000001 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.159511+0000", 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4607e-05 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.159766+0000", 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025409800000000002 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162941+0000", 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031756779999999999 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164046+0000", 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011049860000000001 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164201+0000", 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015452099999999999 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164318+0000", 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011774500000000001 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.453 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6977 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152257+0000", 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.1890396999999999, 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012224376, 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6977 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152257+0000", 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152257+0000", 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152258+0000", 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152265+0000", 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8600000000000004e-06 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152281+0000", 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6534000000000001e-05 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152298+0000", 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6665999999999999e-05 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.160243+0000", 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079454250000000008 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.160308+0000", 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4622000000000006e-05 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.160554+0000", 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024636599999999998 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162962+0000", 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024072389999999998 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164391+0000", 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014290259999999999 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164429+0000", 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8683999999999998e-05 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164481+0000", 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1753999999999999e-05 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:23.459 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:23.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:23.461 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:23.461+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:23.562 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:23.561+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:23.593 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:23.594 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:23.594 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:23.594 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:23.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:23.662 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:23.660+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:23.735 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:23.735 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:23.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:23.793 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:23.792+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:23.864 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:23.861+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.309555781999997, 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.036907083, 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:27:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:27:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.0368719070000001, 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:27:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.036783228, 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.0366812850000002, 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:27:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:27:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.0365721829999996, 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:27:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.0364743240000003, 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:27:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:27:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.0364092740000004, 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:27:23.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.0363166279999998, 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.964 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:23.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:27:23.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:27:23.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.0362446240000001, 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.970 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:27:23.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2767831379999999, 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.976 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:23.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:27:23.981 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2767638899999998, 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.982 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:27:23.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:27:23.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2767137699999997, 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.988 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:27:23.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:23.993 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2766927189999997, 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:23.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:27:23.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:27:23.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:23.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2766773999999996, 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:24.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:27:24.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:24.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:24.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2766605179999999, 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.005 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:24.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:24.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.010 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:24.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2766421819999998, 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:24.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:27:24.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:27:24.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:24.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.019 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2766261800000001, 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:24.020 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:27:24.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:27:24.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:24.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2765398169999997, 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.026 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:27:24.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:27:24.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:27:24.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:24.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:24.031 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2665773409999996, 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:24.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:24.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:27:24.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:24.037 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:24.038 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:24.039 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:24.037+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:24.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:24.131 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone11 from 0xafef800 (overlap 0xafef800) to 0x0 2024-05-06T00:27:24.142 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:24.140+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:24.195 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:24.195 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:24.195 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:24.195 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:24.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:24.241 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:24.237+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:24.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:24.253 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:24.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:24.286 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:24.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:24.317 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:24.341 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:24.341+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:24.352 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:24.352 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:24.379 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:24.379 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:24.442 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:24.440+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:24.449 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:24.449 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:24.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:24.552 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:24.552+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:24.629 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:24.629 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:24.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:24.644 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:24.641+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:24.744 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:24.740+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:24.813 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:24.814 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:24.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:24.873 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:24.872+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:24.946 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:24.945+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:24.948 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-05-06T00:27:24.995 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:27:25.001 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:25.001 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:25.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:25.024 INFO:teuthology.orchestra.run.smithi186.stdout:1692 punch from 0x69c9600 to 0x69cbe00, (0x2800 bytes) 2024-05-06T00:27:25.042 INFO:teuthology.orchestra.run.smithi186.stdout:1697 read 0x8c5f400 thru 0x8c61a5a (0x265b bytes) 2024-05-06T00:27:25.042 INFO:teuthology.orchestra.run.smithi186.stdout:1699 write 0xad2b800 thru 0xad33c4c (0x844d bytes) 2024-05-06T00:27:25.043 INFO:teuthology.orchestra.run.smithi186.stdout:1700 write 0x1595600 thru 0x15a2286 (0xcc87 bytes) 2024-05-06T00:27:25.045 INFO:teuthology.orchestra.run.smithi186.stdout:1702 punch from 0xc38800 to 0xc3de00, (0x5600 bytes) 2024-05-06T00:27:25.049 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:25.045+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:25.143 INFO:teuthology.orchestra.run.smithi186.stdout:1703 write 0x2354800 thru 0x2357b65 (0x3366 bytes) 2024-05-06T00:27:25.144 INFO:teuthology.orchestra.run.smithi186.stdout:1704 read 0xaf1a400 thru 0xaf214e7 (0x70e8 bytes) 2024-05-06T00:27:25.148 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:25.144+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:25.152 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:25.152 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:25.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:25.186 INFO:teuthology.orchestra.run.smithi186.stdout:1706 trunc from 0xafef800 to 0x81ffc00 2024-05-06T00:27:25.250 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:25.249+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:25.288 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:25.288 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:25.302 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:25.355 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:25.348+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:25.371 INFO:teuthology.orchestra.run.smithi186.stdout:1709 read 0x7638a00 thru 0x763da65 (0x5066 bytes) 2024-05-06T00:27:25.380 INFO:teuthology.orchestra.run.smithi186.stdout:1712 read 0x5f82a00 thru 0x5f8e515 (0xbb16 bytes) 2024-05-06T00:27:25.380 INFO:teuthology.orchestra.run.smithi186.stdout:1716 write 0x5ec4c00 thru 0x5ec75aa (0x29ab bytes) 2024-05-06T00:27:25.425 INFO:teuthology.orchestra.run.smithi186.stdout:1717 write 0xc138400 thru 0xc13fab6 (0x76b7 bytes) 2024-05-06T00:27:25.428 INFO:teuthology.orchestra.run.smithi186.stdout:1718 read 0x475f000 thru 0x475fa2d (0xa2e bytes) 2024-05-06T00:27:25.453 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:25.449+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:25.475 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:25.476 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:25.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:25.497 INFO:teuthology.orchestra.run.smithi186.stdout:1720 write 0xe980800 thru 0xe989b8f (0x9390 bytes) 2024-05-06T00:27:25.500 INFO:teuthology.orchestra.run.smithi186.stdout:1721 read 0x81bcc00 thru 0x81cc35d (0xf75e bytes) 2024-05-06T00:27:25.503 INFO:teuthology.orchestra.run.smithi186.stdout:1723 trunc from 0xe989c00 to 0xe276000 2024-05-06T00:27:25.538 INFO:teuthology.orchestra.run.smithi186.stdout:1725 read 0x837bc00 thru 0x837ff79 (0x437a bytes) 2024-05-06T00:27:25.541 INFO:teuthology.orchestra.run.smithi186.stdout:1726 write 0x28b3e00 thru 0x28c20ba (0xe2bb bytes) 2024-05-06T00:27:25.543 INFO:teuthology.orchestra.run.smithi186.stdout:1728 read 0xd0fba00 thru 0xd10534d (0x994e bytes) 2024-05-06T00:27:25.545 INFO:teuthology.orchestra.run.smithi186.stdout:1731 read 0x8847000 thru 0x884d011 (0x6012 bytes) 2024-05-06T00:27:25.545 INFO:teuthology.orchestra.run.smithi186.stdout:1732 punch from 0x95bc00 to 0x96a200, (0xe600 bytes) 2024-05-06T00:27:25.547 INFO:teuthology.orchestra.run.smithi186.stdout:1733 write 0x25ab200 thru 0x25aec47 (0x3a48 bytes) 2024-05-06T00:27:25.547 INFO:teuthology.orchestra.run.smithi186.stdout:1734 read 0x206200 thru 0x208068 (0x1e69 bytes) 2024-05-06T00:27:25.552 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:25.552+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:25.597 INFO:teuthology.orchestra.run.smithi186.stdout:1737 write 0xe5f8600 thru 0xe5fadd4 (0x27d5 bytes) 2024-05-06T00:27:25.599 INFO:teuthology.orchestra.run.smithi186.stdout:1739 write 0x528b800 thru 0x528cbc9 (0x13ca bytes) 2024-05-06T00:27:25.600 INFO:teuthology.orchestra.run.smithi186.stdout:1740 punch from 0xb84e200 to 0xb858000, (0x9e00 bytes) 2024-05-06T00:27:25.600 INFO:teuthology.orchestra.run.smithi186.stdout:1741 punch from 0x829f200 to 0x82a2e00, (0x3c00 bytes) 2024-05-06T00:27:25.601 INFO:teuthology.orchestra.run.smithi186.stdout:1742 write 0x9168800 thru 0x9178711 (0xff12 bytes) 2024-05-06T00:27:25.601 INFO:teuthology.orchestra.run.smithi186.stdout:1748 read 0x800a600 thru 0x8012f98 (0x8999 bytes) 2024-05-06T00:27:25.602 INFO:teuthology.orchestra.run.smithi186.stdout:1749 write 0x5b5e000 thru 0x5b67fc2 (0x9fc3 bytes) 2024-05-06T00:27:25.603 INFO:teuthology.orchestra.run.smithi186.stdout:1751 read 0x81c8200 thru 0x81cb38b (0x318c bytes) 2024-05-06T00:27:25.604 INFO:teuthology.orchestra.run.smithi186.stdout:1752 read 0xb4ad600 thru 0xb4ba2ab (0xccac bytes) 2024-05-06T00:27:25.605 INFO:teuthology.orchestra.run.smithi186.stdout:1753 write 0x3b47a00 thru 0x3b5217f (0xa780 bytes) 2024-05-06T00:27:25.605 INFO:teuthology.orchestra.run.smithi186.stdout:1754 write 0xae63800 thru 0xae72180 (0xe981 bytes) 2024-05-06T00:27:25.606 INFO:teuthology.orchestra.run.smithi186.stdout:1758 write 0xc509400 thru 0xc512505 (0x9106 bytes) 2024-05-06T00:27:25.609 INFO:teuthology.orchestra.run.smithi186.stdout:1761 trunc from 0xe5fae00 to 0xca29e00 2024-05-06T00:27:25.653 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:25.653+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:25.661 INFO:teuthology.orchestra.run.smithi186.stdout:1763 read 0x731a200 thru 0x7323cf6 (0x9af7 bytes) 2024-05-06T00:27:25.664 INFO:teuthology.orchestra.run.smithi186.stdout:1764 read 0x7ed4e00 thru 0x7ed5994 (0xb95 bytes) 2024-05-06T00:27:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:25.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:25.722 INFO:teuthology.orchestra.run.smithi186.stdout:1765 trunc from 0xca29e00 to 0x4cdfe00 2024-05-06T00:27:25.767 INFO:teuthology.orchestra.run.smithi186.stdout:1766 read 0x3836200 thru 0x38382da (0x20db bytes) 2024-05-06T00:27:25.771 INFO:teuthology.orchestra.run.smithi186.stdout:1767 punch from 0x512000 to 0x521c00, (0xfc00 bytes) 2024-05-06T00:27:25.773 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:25.774+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:25.776 INFO:teuthology.orchestra.run.smithi186.stdout:1768 punch from 0x3d07000 to 0x3d0ae00, (0x3e00 bytes) 2024-05-06T00:27:25.778 INFO:teuthology.orchestra.run.smithi186.stdout:1770 read 0x25db200 thru 0x25e5116 (0x9f17 bytes) 2024-05-06T00:27:25.779 INFO:teuthology.orchestra.run.smithi186.stdout:1771 read 0x19cf600 thru 0x19d048b (0xe8c bytes) 2024-05-06T00:27:25.816 INFO:teuthology.orchestra.run.smithi186.stdout:1772 trunc from 0x4cdfe00 to 0x26a2e00 2024-05-06T00:27:25.859 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:25.852+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:25.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:25.891 INFO:teuthology.orchestra.run.smithi186.stdout:1774 read 0x1a82600 thru 0x1a83361 (0xd62 bytes) 2024-05-06T00:27:25.894 INFO:teuthology.orchestra.run.smithi186.stdout:1777 read 0x1eea000 thru 0x1ef2132 (0x8133 bytes) 2024-05-06T00:27:25.935 INFO:teuthology.orchestra.run.smithi186.stdout:1779 write 0x9d7ee00 thru 0x9d88a66 (0x9c67 bytes) 2024-05-06T00:27:25.937 INFO:teuthology.orchestra.run.smithi186.stdout:1781 punch from 0x7945a00 to 0x7949000, (0x3600 bytes) 2024-05-06T00:27:25.938 INFO:teuthology.orchestra.run.smithi186.stdout:1783 write 0x3087a00 thru 0x30926f8 (0xacf9 bytes) 2024-05-06T00:27:25.940 INFO:teuthology.orchestra.run.smithi186.stdout:1784 write 0x3929e00 thru 0x3934fb5 (0xb1b6 bytes) 2024-05-06T00:27:25.941 INFO:teuthology.orchestra.run.smithi186.stdout:1785 punch from 0x4f6a200 to 0x4f73200, (0x9000 bytes) 2024-05-06T00:27:25.942 INFO:teuthology.orchestra.run.smithi186.stdout:1786 read 0x1b21e00 thru 0x1b2aba6 (0x8da7 bytes) 2024-05-06T00:27:25.957 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:25.956+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:25.974 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:25.974 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-05-06T00:27:25.976 INFO:teuthology.orchestra.run.smithi186.stdout:1787 write 0xe5f6c00 thru 0xe6068bc (0xfcbd bytes) 2024-05-06T00:27:25.994 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:27:25.998 INFO:teuthology.orchestra.run.smithi186.stdout:1791 trunc from 0xe606a00 to 0x3013000 2024-05-06T00:27:26.059 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:26.056+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:26.066 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:26.066 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:26.074 INFO:teuthology.orchestra.run.smithi186.stdout:1793 write 0x3575000 thru 0x3575e2c (0xe2d bytes) 2024-05-06T00:27:26.076 INFO:teuthology.orchestra.run.smithi186.stdout:1794 write 0x21bac00 thru 0x21c4d33 (0xa134 bytes) 2024-05-06T00:27:26.078 INFO:teuthology.orchestra.run.smithi186.stdout:1797 read 0x1b0d400 thru 0x1b15068 (0x7c69 bytes) 2024-05-06T00:27:26.079 INFO:teuthology.orchestra.run.smithi186.stdout:1798 punch from 0x24b000 to 0x24b800, (0x800 bytes) 2024-05-06T00:27:26.082 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:26.111 INFO:teuthology.orchestra.run.smithi186.stdout:1799 write 0x7fc2a00 thru 0x7fcbbd5 (0x91d6 bytes) 2024-05-06T00:27:26.113 INFO:teuthology.orchestra.run.smithi186.stdout:1800 write 0x1e6c00 thru 0x1eabac (0x3fad bytes) 2024-05-06T00:27:26.121 INFO:teuthology.orchestra.run.smithi186.stdout:1801 trunc from 0x7fcbc00 to 0x40d2200 2024-05-06T00:27:26.160 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:26.160+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:26.201 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:26.201 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:26.210 INFO:teuthology.orchestra.run.smithi186.stdout:1803 write 0xe609c00 thru 0xe61120d (0x760e bytes) 2024-05-06T00:27:26.212 INFO:teuthology.orchestra.run.smithi186.stdout:1805 punch from 0xb21ce00 to 0xb227400, (0xa600 bytes) 2024-05-06T00:27:26.213 INFO:teuthology.orchestra.run.smithi186.stdout:1806 read 0x288ac00 thru 0x28922d6 (0x76d7 bytes) 2024-05-06T00:27:26.213 INFO:teuthology.orchestra.run.smithi186.stdout:1807 read 0x5484800 thru 0x5486408 (0x1c09 bytes) 2024-05-06T00:27:26.213 INFO:teuthology.orchestra.run.smithi186.stdout:1809 punch from 0x6e56600 to 0x6e64200, (0xdc00 bytes) 2024-05-06T00:27:26.214 INFO:teuthology.orchestra.run.smithi186.stdout:1810 punch from 0x241d000 to 0x241da00, (0xa00 bytes) 2024-05-06T00:27:26.214 INFO:teuthology.orchestra.run.smithi186.stdout:1812 read 0xb7d1400 thru 0xb7d8522 (0x7123 bytes) 2024-05-06T00:27:26.214 INFO:teuthology.orchestra.run.smithi186.stdout:1813 read 0x5091e00 thru 0x509b98a (0x9b8b bytes) 2024-05-06T00:27:26.214 INFO:teuthology.orchestra.run.smithi186.stdout:1814 write 0xacd0e00 thru 0xacd43ab (0x35ac bytes) 2024-05-06T00:27:26.215 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:26.229 INFO:teuthology.orchestra.run.smithi186.stdout:1816 trunc from 0xe611400 to 0x5939e00 2024-05-06T00:27:26.262 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:26.262+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:26.301 INFO:teuthology.orchestra.run.smithi186.stdout:1817 write 0xb8e4000 thru 0xb8e7cdc (0x3cdd bytes) 2024-05-06T00:27:26.302 INFO:teuthology.orchestra.run.smithi186.stdout:1818 write 0x9809600 thru 0x980c141 (0x2b42 bytes) 2024-05-06T00:27:26.302 INFO:teuthology.orchestra.run.smithi186.stdout:1820 read 0x71a0e00 thru 0x71a9fc6 (0x91c7 bytes) 2024-05-06T00:27:26.303 INFO:teuthology.orchestra.run.smithi186.stdout:1825 punch from 0x95e9800 to 0x95ed800, (0x4000 bytes) 2024-05-06T00:27:26.303 INFO:teuthology.orchestra.run.smithi186.stdout:1826 punch from 0x2e1fe00 to 0x2e2ec00, (0xee00 bytes) 2024-05-06T00:27:26.304 INFO:teuthology.orchestra.run.smithi186.stdout:1827 trunc from 0xb8e7e00 to 0xbad1c00 2024-05-06T00:27:26.334 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:26.334 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:26.338 INFO:teuthology.orchestra.run.smithi186.stdout:1828 read 0x5efc00 thru 0x5f17d1 (0x1bd2 bytes) 2024-05-06T00:27:26.344 INFO:teuthology.orchestra.run.smithi186.stdout:1829 trunc from 0xbad1c00 to 0xab2c600 2024-05-06T00:27:26.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:26.363 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:26.360+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:26.368 INFO:teuthology.orchestra.run.smithi186.stdout:1833 read 0x64ec600 thru 0x64fa3b8 (0xddb9 bytes) 2024-05-06T00:27:26.368 INFO:teuthology.orchestra.run.smithi186.stdout:1834 read 0x27f5400 thru 0x27f965a (0x425b bytes) 2024-05-06T00:27:26.369 INFO:teuthology.orchestra.run.smithi186.stdout:1837 read 0x3315a00 thru 0x3321cb7 (0xc2b8 bytes) 2024-05-06T00:27:26.370 INFO:teuthology.orchestra.run.smithi186.stdout:1838 write 0x8999600 thru 0x89a5af0 (0xc4f1 bytes) 2024-05-06T00:27:26.371 INFO:teuthology.orchestra.run.smithi186.stdout:1839 read 0x15b6800 thru 0x15ba0ff (0x3900 bytes) 2024-05-06T00:27:26.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:26.405 INFO:teuthology.orchestra.run.smithi186.stdout:1841 write 0xd6dfe00 thru 0xd6e576f (0x5970 bytes) 2024-05-06T00:27:26.407 INFO:teuthology.orchestra.run.smithi186.stdout:1842 write 0x10a3e00 thru 0x10a4219 (0x41a bytes) 2024-05-06T00:27:26.422 INFO:teuthology.orchestra.run.smithi186.stdout:1843 trunc from 0xd6e5800 to 0x75a1600 2024-05-06T00:27:26.465 INFO:teuthology.orchestra.run.smithi186.stdout:1844 punch from 0x22ae200 to 0x22b7a00, (0x9800 bytes) 2024-05-06T00:27:26.468 INFO:teuthology.orchestra.run.smithi186.stdout:1846 read 0x64d4c00 thru 0x64da0aa (0x54ab bytes) 2024-05-06T00:27:26.477 INFO:teuthology.orchestra.run.smithi186.stdout:1847 trunc from 0x75a1600 to 0xb617400 2024-05-06T00:27:26.481 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:26.482+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:26.498 INFO:teuthology.orchestra.run.smithi186.stdout:1848 read 0xcb0e00 thru 0xcb85c4 (0x77c5 bytes) 2024-05-06T00:27:26.519 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:26.520 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:26.527 INFO:teuthology.orchestra.run.smithi186.stdout:1850 write 0xd6fc600 thru 0xd701172 (0x4b73 bytes) 2024-05-06T00:27:26.528 INFO:teuthology.orchestra.run.smithi186.stdout:1851 write 0x7f9ba00 thru 0x7f9fcec (0x42ed bytes) 2024-05-06T00:27:26.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:26.542 INFO:teuthology.orchestra.run.smithi186.stdout:1853 trunc from 0xd701200 to 0x88ca000 2024-05-06T00:27:26.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:26.577 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:26.578 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:26.574+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:26.589 INFO:teuthology.orchestra.run.smithi186.stdout:1854 write 0x751aa00 thru 0x75284f0 (0xdaf1 bytes) 2024-05-06T00:27:26.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:26.593 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:26.599 INFO:teuthology.orchestra.run.smithi186.stdout:1855 trunc from 0x88ca000 to 0x622c600 2024-05-06T00:27:26.610 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:26.611 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:26.627 INFO:teuthology.orchestra.run.smithi186.stdout:1856 punch from 0x7d9e00 to 0x7e9000, (0xf200 bytes) 2024-05-06T00:27:26.630 INFO:teuthology.orchestra.run.smithi186.stdout:1857 read 0x11b200 thru 0x126c3f (0xba40 bytes) 2024-05-06T00:27:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:26.631 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:26.653 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:26.653 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:26.666 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:26.664+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:26.668 INFO:teuthology.orchestra.run.smithi186.stdout:1860 write 0xbd63800 thru 0xbd6e43b (0xac3c bytes) 2024-05-06T00:27:26.670 INFO:teuthology.orchestra.run.smithi186.stdout:1864 read 0x6c4a400 thru 0x6c542c6 (0x9ec7 bytes) 2024-05-06T00:27:26.670 INFO:teuthology.orchestra.run.smithi186.stdout:1865 punch from 0x591bc00 to 0x5924e00, (0x9200 bytes) 2024-05-06T00:27:26.671 INFO:teuthology.orchestra.run.smithi186.stdout:1867 write 0xb0ae400 thru 0xb0b836d (0x9f6e bytes) 2024-05-06T00:27:26.671 INFO:teuthology.orchestra.run.smithi186.stdout:1873 read 0x33e4a00 thru 0x33e4e2a (0x42b bytes) 2024-05-06T00:27:26.675 INFO:teuthology.orchestra.run.smithi186.stdout:1874 read 0x2c59a00 thru 0x2c5fc8c (0x628d bytes) 2024-05-06T00:27:26.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:26.675 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:26.676 INFO:teuthology.orchestra.run.smithi186.stdout:1879 read 0x6c32c00 thru 0x6c37fa9 (0x53aa bytes) 2024-05-06T00:27:26.676 INFO:teuthology.orchestra.run.smithi186.stdout:1881 write 0xa98ce00 thru 0xa995123 (0x8324 bytes) 2024-05-06T00:27:26.676 INFO:teuthology.orchestra.run.smithi186.stdout:1883 punch from 0xbd04600 to 0xbd0a600, (0x6000 bytes) 2024-05-06T00:27:26.677 INFO:teuthology.orchestra.run.smithi186.stdout:1887 read 0x2422200 thru 0x242ff82 (0xdd83 bytes) 2024-05-06T00:27:26.682 INFO:teuthology.orchestra.run.smithi186.stdout:1888 trunc from 0xbd6e600 to 0xaacb400 2024-05-06T00:27:26.702 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:26.702 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:26.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:26.702 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:26.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.702 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2567 3.1e 3:7ff04bbe:::rbd_data.110cc8beeb04.0000000000000005:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-06T00:27:26.703 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:26.703 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.203036565000001, 2024-05-06T00:27:26.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091482830000000001, 2024-05-06T00:27:26.703 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2567 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498655+0000", 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498657+0000", 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.75e-06 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498666+0000", 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3030000000000005e-06 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498690+0000", 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-05 2024-05-06T00:27:26.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.498707+0000", 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7326999999999999e-05 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501155+0000", 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002447884 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501468+0000", 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031281100000000002 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.501983+0000", 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051484999999999996 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.505388+0000", 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034055240000000001 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507550+0000", 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021622690000000001 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507694+0000", 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014377899999999999 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.507804+0000", 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109486 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1779 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.045576519000001, 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095752939999999998, 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1779 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656115+0000", 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656116+0000", 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9699999999999995e-07 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656120+0000", 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5939999999999998e-06 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656128+0000", 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0309999999999998e-06 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.656133+0000", 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.659748+0000", 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003615231 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.659987+0000", 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023881900000000001 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.660658+0000", 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067093800000000005 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.663069+0000", 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024110109999999998 2024-05-06T00:27:26.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665388+0000", 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023186579999999999 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665608+0000", 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022027 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.665691+0000", 2024-05-06T00:27:26.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2698e-05 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1841 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.016238938000001, 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011157446999999999, 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.715 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1841 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685453+0000", 2024-05-06T00:27:26.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685454+0000", 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2200000000000002e-07 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685459+0000", 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4820000000000002e-06 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685469+0000", 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8020000000000003e-06 2024-05-06T00:27:26.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.685476+0000", 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-06 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.689403+0000", 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003927346 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.689582+0000", 2024-05-06T00:27:26.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017826300000000001 2024-05-06T00:27:26.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.690411+0000", 2024-05-06T00:27:26.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00082928899999999998 2024-05-06T00:27:26.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696488+0000", 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060775960000000002 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696529+0000", 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0416000000000001e-05 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696571+0000", 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1931000000000002e-05 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696610+0000", 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9660000000000003e-05 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1848 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.014753631, 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011575561999999999, 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1848 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.723 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686938+0000", 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686939+0000", 2024-05-06T00:27:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.136e-06 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686946+0000", 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2790000000000004e-06 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686959+0000", 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3774000000000001e-05 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.686969+0000", 2024-05-06T00:27:26.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2070000000000001e-06 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.691625+0000", 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046563450000000001 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.691714+0000", 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8651000000000002e-05 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692727+0000", 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010135400000000001 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696654+0000", 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039264529999999999 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698290+0000", 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016358880000000001 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698414+0000", 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000124082 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698514+0000", 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100207 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1849 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.728 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.014539506, 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013152664, 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1849 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687152+0000", 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687153+0000", 2024-05-06T00:27:26.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6599999999999994e-07 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687172+0000", 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8868999999999998e-05 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687187+0000", 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4491999999999999e-05 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687196+0000", 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6039999999999999e-06 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.694179+0000", 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069822119999999998 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.694396+0000", 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000217856 2024-05-06T00:27:26.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.695435+0000", 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001038349 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700028+0000", 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045936090000000002 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700067+0000", 2024-05-06T00:27:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9048999999999998e-05 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700203+0000", 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013551700000000001 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700305+0000", 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102141 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1850 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.013909343, 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01443141, 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1850 2024-05-06T00:27:26.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687783+0000", 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687784+0000", 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.311e-06 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687789+0000", 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5289999999999999e-06 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687809+0000", 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9911e-05 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.687818+0000", 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9549999999999998e-06 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696755+0000", 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089369069999999991 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.696977+0000", 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000221337 2024-05-06T00:27:26.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.697535+0000", 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00055843500000000003 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700358+0000", 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028226850000000001 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702017+0000", 2024-05-06T00:27:26.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001659537 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702117+0000", 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9984999999999999e-05 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702214+0000", 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6817999999999994e-05 2024-05-06T00:27:26.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1869 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.009483757, 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011357313000000001, 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.741 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1869 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692208+0000", 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692210+0000", 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.514e-06 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692219+0000", 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2399999999999996e-06 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692240+0000", 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1537000000000001e-05 2024-05-06T00:27:26.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692257+0000", 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6670000000000001e-05 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698589+0000", 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063314920000000002 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.698673+0000", 2024-05-06T00:27:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4358999999999996e-05 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.699065+0000", 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039182699999999997 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.702254+0000", 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031890780000000001 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703473+0000", 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012188730000000001 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703526+0000", 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3588999999999998e-05 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703565+0000", 2024-05-06T00:27:26.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9134000000000002e-05 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:1870 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.009379751999999, 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011641416999999999, 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1870 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692312+0000", 2024-05-06T00:27:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692314+0000", 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.42e-06 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692323+0000", 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1409999999999994e-06 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692346+0000", 2024-05-06T00:27:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2955999999999998e-05 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.692363+0000", 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6923999999999999e-05 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700432+0000", 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080692569999999998 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.700598+0000", 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166422 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.701369+0000", 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00077104099999999998 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703579+0000", 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022098 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703712+0000", 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133378 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703863+0000", 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015091700000000001 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.703954+0000", 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0161000000000006e-05 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.752 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2082 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.887650485, 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098291899999999998, 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2082 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:26.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814041+0000", 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814043+0000", 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2720000000000001e-06 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814049+0000", 2024-05-06T00:27:26.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814061+0000", 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1815e-05 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814069+0000", 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4630000000000001e-06 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.816625+0000", 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025561239999999999 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.816767+0000", 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014136199999999999 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.817683+0000", 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091655999999999997 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823412+0000", 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057287509999999998 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823578+0000", 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016635 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823763+0000", 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184172 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823871+0000", 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108068 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2083 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.887555849, 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099999149999999998, 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2083 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814136+0000", 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814137+0000", 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9299999999999997e-07 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814141+0000", 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5750000000000002e-06 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814151+0000", 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4450000000000002e-06 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814160+0000", 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3279999999999995e-06 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.818686+0000", 2024-05-06T00:27:26.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045255809999999999 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.818863+0000", 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017779199999999999 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.819847+0000", 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098399199999999994 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823546+0000", 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036989969999999999 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.823898+0000", 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035146899999999998 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824030+0000", 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000132392 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824136+0000", 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105651 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4619.0:2084 3.7 3:e4ad76af:::rbd_object_map.110cd55b09e5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.887468455, 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011218071, 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.765 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4619", 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786269007", 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2084 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814223+0000", 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814225+0000", 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.13e-06 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814230+0000", 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1150000000000002e-06 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814243+0000", 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3101999999999999e-05 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.814250+0000", 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8759999999999999e-06 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.821215+0000", 2024-05-06T00:27:26.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069655439999999997 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.821467+0000", 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025156400000000002 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.822255+0000", 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00078842500000000004 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.824161+0000", 2024-05-06T00:27:26.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019054429999999999 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825286+0000", 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001125687 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825375+0000", 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9013999999999999e-05 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:12.825442+0000", 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6171000000000001e-05 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4646.0:109 3.8 3:11ca53b4:::rbd_object_map.110cc6eb3028:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.144353859000001, 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010787722, 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4646", 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/4090392681", 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 109 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557338+0000", 2024-05-06T00:27:26.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557341+0000", 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.027e-06 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557357+0000", 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5926999999999998e-05 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557392+0000", 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5264e-05 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.557414+0000", 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1557e-05 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.561705+0000", 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042916580000000003 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.561768+0000", 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2188999999999997e-05 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.562103+0000", 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033499199999999999 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.567706+0000", 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056037730000000003 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568002+0000", 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029544 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568076+0000", 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3796999999999998e-05 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:14.568126+0000", 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0098e-05 2024-05-06T00:27:26.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4863.0:3623 3.d e43/18 3:b17d50d3:::rbd_object_map.110cf57807c6:head v 43'218, mlcod=43'208)", 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.9272566480000002, 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097994039999999994, 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774435+0000", 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774437+0000", 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774450+0000", 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3257e-05 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774476+0000", 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6108000000000001e-05 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.774496+0000", 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9843e-05 2024-05-06T00:27:26.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.777187+0000", 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026909 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.777241+0000", 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4104000000000002e-05 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784162+0000", 2024-05-06T00:27:26.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069210519999999996 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:18.784235+0000", 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2596999999999996e-05 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6974 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152092+0000", 2024-05-06T00:27:26.781 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5495994309999999, 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010723718, 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6974 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.782 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152092+0000", 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152092+0000", 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152093+0000", 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4700000000000001e-07 2024-05-06T00:27:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152097+0000", 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.216e-06 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152107+0000", 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2469999999999997e-06 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152114+0000", 2024-05-06T00:27:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3300000000000001e-06 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.157905+0000", 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005791175 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.157965+0000", 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9505e-05 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.158220+0000", 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025549500000000001 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162593+0000", 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043727949999999996 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162689+0000", 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6514000000000005e-05 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162753+0000", 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4015000000000003e-05 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162816+0000", 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2779000000000003e-05 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6975 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152139+0000", 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5495526670000004, 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011796399000000001, 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6975 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152139+0000", 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152139+0000", 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152140+0000", 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9000000000000005e-07 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152147+0000", 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1729999999999998e-06 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152167+0000", 2024-05-06T00:27:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9734000000000001e-05 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152184+0000", 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6387000000000001e-05 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.158668+0000", 2024-05-06T00:27:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064840829999999999 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.158728+0000", 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0868000000000001e-05 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.159005+0000", 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000276879 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162611+0000", 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003606106 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.163838+0000", 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012269819999999999 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.163881+0000", 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2253999999999999e-05 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.163936+0000", 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4942999999999999e-05 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6976 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152202+0000", 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5494902230000003, 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012116782, 2024-05-06T00:27:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6976 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152202+0000", 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152202+0000", 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152203+0000", 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8000000000000004e-07 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152210+0000", 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2259999999999999e-06 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152226+0000", 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6470999999999999e-05 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152240+0000", 2024-05-06T00:27:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3665999999999999e-05 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.159447+0000", 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072069040000000001 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.159511+0000", 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4607e-05 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.159766+0000", 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025409800000000002 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162941+0000", 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031756779999999999 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164046+0000", 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011049860000000001 2024-05-06T00:27:26.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164201+0000", 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015452099999999999 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164318+0000", 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011774500000000001 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4863.0:6977 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:22.152257+0000", 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5494353060000003, 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012224376, 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4863", 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6977 2024-05-06T00:27:26.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152257+0000", 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152257+0000", 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152258+0000", 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152265+0000", 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8600000000000004e-06 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152281+0000", 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6534000000000001e-05 2024-05-06T00:27:26.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.152298+0000", 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6665999999999999e-05 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.160243+0000", 2024-05-06T00:27:26.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079454250000000008 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.160308+0000", 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4622000000000006e-05 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:26.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.160554+0000", 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024636599999999998 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.162962+0000", 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024072389999999998 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164391+0000", 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014290259999999999 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164429+0000", 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8683999999999998e-05 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:22.164481+0000", 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1753999999999999e-05 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.807 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:282 3.18 3:1af02675:::rbd_data.110cd5ddb45c.0000000000000002:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.141118+0000", 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.560573953, 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.039756291999999999, 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 282 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141118+0000", 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141118+0000", 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141122+0000", 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7859999999999998e-06 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141137+0000", 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4907e-05 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141186+0000", 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9620999999999998e-05 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141209+0000", 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2419999999999999e-05 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141381+0000", 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172189 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141700+0000", 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031862600000000002 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.180874+0000", 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.039174742999999998 2024-05-06T00:27:26.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:284 3.8 3:145217c3:::rbd_data.110cd5ddb45c.0000000000000003:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.141269+0000", 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.5604224379999998, 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046233225000000003, 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 284 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141269+0000", 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141269+0000", 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141271+0000", 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.497e-06 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141281+0000", 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7059999999999999e-06 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141303+0000", 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2413999999999999e-05 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141320+0000", 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141501+0000", 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000181719 2024-05-06T00:27:26.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141812+0000", 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031029300000000001 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187503+0000", 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.045691009999999997 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.816 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:2109 3.8 3:145217c3:::rbd_data.110cd5ddb45c.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.238439+0000", 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.4632529870000002, 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010042087, 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2109 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.238439+0000", 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.238439+0000", 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.238439+0000", 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.238442+0000", 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5399999999999998e-06 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:26.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.238447+0000", 2024-05-06T00:27:26.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6060000000000002e-06 2024-05-06T00:27:26.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:26.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.238452+0000", 2024-05-06T00:27:26.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1989999999999999e-06 2024-05-06T00:27:26.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:26.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.239270+0000", 2024-05-06T00:27:26.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00081787600000000004 2024-05-06T00:27:26.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:26.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.239342+0000", 2024-05-06T00:27:26.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2806999999999996e-05 2024-05-06T00:27:26.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:26.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.239530+0000", 2024-05-06T00:27:26.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018772799999999999 2024-05-06T00:27:26.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:26.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.240521+0000", 2024-05-06T00:27:26.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000990758 2024-05-06T00:27:26.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.248394+0000", 2024-05-06T00:27:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078729639999999997 2024-05-06T00:27:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.248443+0000", 2024-05-06T00:27:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9113999999999998e-05 2024-05-06T00:27:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.248481+0000", 2024-05-06T00:27:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8093999999999998e-05 2024-05-06T00:27:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.826 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:26.826 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:26.826 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:26.829 INFO:teuthology.orchestra.run.smithi186.stdout:1889 punch from 0x785e800 to 0x786b600, (0xce00 bytes) 2024-05-06T00:27:26.829 INFO:teuthology.orchestra.run.smithi186.stdout:1891 read 0x905b600 thru 0x9067287 (0xbc88 bytes) 2024-05-06T00:27:26.830 INFO:teuthology.orchestra.run.smithi186.stdout:1892 trunc from 0xaacb400 to 0x7808600 2024-05-06T00:27:26.830 INFO:teuthology.orchestra.run.smithi186.stdout:1894 write 0x6128a00 thru 0x6137589 (0xeb8a bytes) 2024-05-06T00:27:26.830 INFO:teuthology.orchestra.run.smithi186.stdout:1895 clone 13 order 21 su 32768 sc 10 2024-05-06T00:27:26.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:26.831 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:26.824+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:26.929 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:26.926+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:26.960 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:26.960 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:26.960 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:26.961 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:26.974 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:27.009 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.0 to 1 (10000) 2024-05-06T00:27:27.030 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:27.030+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:27.034 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-05-06T00:27:27.103 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:27.103 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:27.103 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:27.103 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:27.103 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:27.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:27.165 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:27.164+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:27.168 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone12 from 0x7808600 (overlap 0x26a2e00) to 0x22c5c00 2024-05-06T00:27:27.232 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:27.230+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:27.253 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:27.253 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.649208797999997, 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:27.254 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:27.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:27.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3765600990000006, 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:27:27.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:27:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:27:27.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.268 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.268 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:27.268 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3765249229999998, 2024-05-06T00:27:27.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:27:27.268 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.268 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.268 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:27:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:27:27.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:27:27.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3764362440000006, 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:27:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:27:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.376334301, 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:27:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:27:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3762251990000003, 2024-05-06T00:27:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:27:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:27:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:27:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.37612734, 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:27:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:27:27.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:27:27.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3760622900000001, 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:27:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:27:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3759696439999995, 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3758976399999998, 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:27:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:27:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:27:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:27:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6164361540000005, 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:27:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:27:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6164169059999995, 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:27:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:27:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6163667860000004, 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6163457349999995, 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.334 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6163304160000003, 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:27:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:27:27.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6163135339999997, 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:27.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:27:27.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:27:27.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:27:27.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6162951979999995, 2024-05-06T00:27:27.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:27:27.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:27:27.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6162791960000007, 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:27.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:27:27.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6161928329999995, 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.363 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:27:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:27:27.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.368 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.6062303569999994, 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:27:27.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:27:27.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.374 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:27.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:27.376 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:27.372+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:27.477 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:27.478+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:27.548 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:27.549 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:27.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:27.609 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:27.608+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:27.678 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:27.676+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:27.690 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:27.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:27.690 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:27.690 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:27.690 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:27.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:27.761 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-05-06T00:27:27.793 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:27.792+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:27.871 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2163 3.a 3:56be37f1:::rbd_data.110cd5ddb45c.000000000000005b:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.239632+0000", 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6301287479999997, 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025379202, 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2163 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.239632+0000", 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.239632+0000", 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.239633+0000", 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1699999999999997e-07 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.239636+0000", 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3720000000000001e-06 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.239645+0000", 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1120000000000002e-06 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.239652+0000", 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7499999999999997e-06 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.242886+0000", 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003233872 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.242996+0000", 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110086 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.265011+0000", 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022015192999999999 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2184 3.11 3:8d4e0d24:::rbd_data.110cd5ddb45c.0000000000000030:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.241829+0000", 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.627931749, 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021329987000000002, 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2184 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.241829+0000", 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.241829+0000", 2024-05-06T00:27:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.241830+0000", 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.241836+0000", 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6540000000000004e-06 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.241847+0000", 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0842999999999999e-05 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.241854+0000", 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5009999999999998e-06 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.241946+0000", 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1984999999999994e-05 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.242059+0000", 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112359 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.242264+0000", 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020596300000000001 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.247147+0000", 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048827539999999996 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.263071+0000", 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015923969999999999 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.263121+0000", 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0164999999999998e-05 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.263159+0000", 2024-05-06T00:27:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7642999999999999e-05 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2744 3.1a 3:58733684:::rbd_data.110cd5ddb45c.0000000000000071:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.269357+0000", 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6004041459999998, 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019662852000000001, 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2744 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269357+0000", 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269357+0000", 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269357+0000", 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1200000000000003e-07 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269360+0000", 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9340000000000002e-06 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269373+0000", 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2779000000000001e-05 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269388+0000", 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4832e-05 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285413+0000", 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016025691000000002 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285482+0000", 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8477000000000002e-05 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285659+0000", 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017725600000000001 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.288244+0000", 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025844869999999999 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.288891+0000", 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00064757400000000004 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.288981+0000", 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9605999999999996e-05 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.289019+0000", 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8603999999999998e-05 2024-05-06T00:27:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2746 3.a 3:513561cd:::rbd_data.110cd5ddb45c.000000000000008f:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.269397+0000", 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.600364087, 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026342339999999999, 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2746 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269397+0000", 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269397+0000", 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269397+0000", 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2900000000000003e-07 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269400+0000", 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.61e-06 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269407+0000", 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8970000000000002e-06 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.269412+0000", 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.66e-06 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.286448+0000", 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017035813 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.286581+0000", 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013249600000000001 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.295739+0000", 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091582350000000007 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2837 3.a 3:502d5df8:::rbd_data.110cd5ddb45c.0000000000000081:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.278444+0000", 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.591316682, 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019397016999999999, 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2837 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.278444+0000", 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.278444+0000", 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.278445+0000", 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2610000000000001e-06 2024-05-06T00:27:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.278454+0000", 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2179999999999999e-06 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.278466+0000", 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.278474+0000", 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9049999999999997e-06 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.287290+0000", 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088163359999999993 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.287366+0000", 2024-05-06T00:27:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6586000000000002e-05 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.287606+0000", 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000239717 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296765+0000", 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091591109999999993 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297735+0000", 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00096977999999999995 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297794+0000", 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9058999999999998e-05 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297841+0000", 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6968000000000002e-05 2024-05-06T00:27:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2983 3.14 3:2834570d:::rbd_data.110cd5ddb45c.00000000000000c9:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.296452+0000", 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5733091420000003, 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022534172000000002, 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2983 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296452+0000", 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296452+0000", 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296452+0000", 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296467+0000", 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4289e-05 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296476+0000", 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3610000000000007e-06 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296483+0000", 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470000000000004e-06 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296543+0000", 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9536000000000001e-05 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296616+0000", 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3021000000000004e-05 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296796+0000", 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018030800000000001 2024-05-06T00:27:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301701+0000", 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004904907 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318898+0000", 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017196587999999999 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318946+0000", 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8476000000000001e-05 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318986+0000", 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9789000000000001e-05 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4932.0:2987 3.1b e49/18 3:dc50fd0a:::rbd_data.110cd5ddb45c.00000000000000c3:head v 49'297, mlcod=49'295)", 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.296859+0000", 2024-05-06T00:27:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5729013520000001, 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021792410000000002, 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296859+0000", 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296859+0000", 2024-05-06T00:27:27.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296860+0000", 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3499999999999995e-07 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296866+0000", 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8340000000000004e-06 2024-05-06T00:27:27.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296875+0000", 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3249999999999997e-06 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296882+0000", 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.635e-06 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296931+0000", 2024-05-06T00:27:27.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9094000000000001e-05 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296947+0000", 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6317e-05 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318614+0000", 2024-05-06T00:27:27.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021666182999999999 2024-05-06T00:27:27.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318652+0000", 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8287000000000001e-05 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2984 3.3 3:c74b8df9:::rbd_data.110cd5ddb45c.00000000000000d5:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.297069+0000", 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.572691378, 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021669753, 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.916 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2984 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297069+0000", 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297069+0000", 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297070+0000", 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7599999999999996e-07 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297078+0000", 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5120000000000002e-06 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297087+0000", 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3610000000000007e-06 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297094+0000", 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7209999999999997e-06 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297150+0000", 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6073000000000003e-05 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297282+0000", 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013193999999999999 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318739+0000", 2024-05-06T00:27:27.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02145737 2024-05-06T00:27:27.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.921 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.921 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2986 3.9 3:94422b3e:::rbd_data.110cd5ddb45c.00000000000000d6:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.297108+0000", 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5726524739999999, 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021826841, 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2986 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297108+0000", 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297108+0000", 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297109+0000", 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7199999999999999e-07 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297111+0000", 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5670000000000002e-06 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297117+0000", 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9020000000000004e-06 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297122+0000", 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6580000000000001e-06 2024-05-06T00:27:27.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297164+0000", 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2002000000000002e-05 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297277+0000", 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113256 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318935+0000", 2024-05-06T00:27:27.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021657984000000002 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(3.1c epoch 49/18 rep_tid 722 entries 49'167 (0'0) error 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head by client.4932.0:2957 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 49'164)", 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.298864+0000", 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.570896308, 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020140247, 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298864+0000", 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298864+0000", 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298865+0000", 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9400000000000004e-07 2024-05-06T00:27:27.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298873+0000", 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1899999999999995e-06 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298906+0000", 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2168000000000001e-05 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298921+0000", 2024-05-06T00:27:27.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5252999999999999e-05 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299002+0000", 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0659000000000001e-05 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319005+0000", 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020003183000000001 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(3.19 epoch 49/18 rep_tid 814 entries 49'100 (0'0) error 3:98dee2cc:::rbd_data.110cd5ddb45c.00000000000000d8:head by client.4932.0:2992 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 49'98)", 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299123+0000", 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5706378779999999, 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019834767, 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299123+0000", 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299123+0000", 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299124+0000", 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6899999999999997e-07 2024-05-06T00:27:27.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299132+0000", 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-06 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299146+0000", 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4032999999999999e-05 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299155+0000", 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5510000000000005e-06 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299226+0000", 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0431000000000004e-05 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318958+0000", 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019731793000000001 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4932.0:2947 3.e e49/18 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head v 49'70, mlcod=49'67)", 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299205+0000", 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5705557479999999, 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019962923, 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299205+0000", 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299205+0000", 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299205+0000", 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299211+0000", 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0289999999999996e-06 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299219+0000", 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1850000000000004e-06 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299224+0000", 2024-05-06T00:27:27.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299284+0000", 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0075999999999999e-05 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299302+0000", 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8283000000000001e-05 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319130+0000", 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019828065999999998 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319168+0000", 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7654999999999998e-05 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4932.0:2949 3.e e49/18 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head v 49'71, mlcod=49'67)", 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299240+0000", 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.570520921, 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019974840000000001, 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299240+0000", 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299240+0000", 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299240+0000", 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299248+0000", 2024-05-06T00:27:27.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1480000000000006e-06 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299254+0000", 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8930000000000002e-06 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299259+0000", 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.391e-06 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299445+0000", 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018609 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299455+0000", 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.798e-06 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319171+0000", 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019716101999999999 2024-05-06T00:27:27.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319215+0000", 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4060000000000002e-05 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4932.0:2956 3.e e49/18 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head v 49'72, mlcod=49'67)", 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299276+0000", 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.57048516, 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019968555999999998, 2024-05-06T00:27:27.943 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299276+0000", 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299276+0000", 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299276+0000", 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1e-07 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299279+0000", 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079999999999999e-06 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299285+0000", 2024-05-06T00:27:27.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299289+0000", 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.873e-06 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299559+0000", 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027035900000000002 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299569+0000", 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4669999999999993e-06 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319217+0000", 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019648399 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319244+0000", 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7095000000000001e-05 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2994 3.a 3:513561cd:::rbd_data.110cd5ddb45c.000000000000008f:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299301+0000", 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5704596469999998, 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019535180999999999, 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2994 2024-05-06T00:27:27.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299301+0000", 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299301+0000", 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299302+0000", 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1500000000000003e-07 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299305+0000", 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1049999999999999e-06 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299312+0000", 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9689999999999997e-06 2024-05-06T00:27:27.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299317+0000", 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3730000000000001e-06 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299469+0000", 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015153200000000001 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299591+0000", 2024-05-06T00:27:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121725 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299891+0000", 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030042300000000002 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.305424+0000", 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005532543 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318740+0000", 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013316766000000001 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318797+0000", 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6558999999999998e-05 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318836+0000", 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9371e-05 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2998 3.15 3:ac11cb60:::rbd_data.110cd5ddb45c.00000000000000c5:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299331+0000", 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5704295789999998, 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019494161999999999, 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2998 2024-05-06T00:27:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299331+0000", 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299331+0000", 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299332+0000", 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3000000000000001e-07 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299335+0000", 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1360000000000001e-06 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299341+0000", 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5950000000000004e-06 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299347+0000", 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7139999999999998e-06 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299407+0000", 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9982000000000003e-05 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299488+0000", 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0839000000000003e-05 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299674+0000", 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018621800000000001 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.302022+0000", 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023474910000000002 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318726+0000", 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016704283 2024-05-06T00:27:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318779+0000", 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3383e-05 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318825+0000", 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5991000000000001e-05 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2999 3.1a 3:5c1cba14:::rbd_data.110cd5ddb45c.00000000000000af:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299362+0000", 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5703986830000001, 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019559712999999999, 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2999 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299362+0000", 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299362+0000", 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299363+0000", 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7000000000000005e-07 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299366+0000", 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1559999999999999e-06 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299373+0000", 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2609999999999996e-06 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299379+0000", 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5459999999999999e-06 2024-05-06T00:27:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300296+0000", 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00091694599999999997 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300399+0000", 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103124 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300596+0000", 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019775700000000001 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.302145+0000", 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015490009999999999 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318847+0000", 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016701185 2024-05-06T00:27:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318890+0000", 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3509e-05 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318922+0000", 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1658000000000001e-05 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3003 3.1a 3:5f1167d8:::rbd_data.110cd5ddb45c.00000000000000b0:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299388+0000", 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.570373247, 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019622825, 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3003 2024-05-06T00:27:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299388+0000", 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299388+0000", 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299388+0000", 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.63e-07 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299393+0000", 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.933e-06 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299399+0000", 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5169999999999996e-06 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299405+0000", 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1499999999999998e-06 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300935+0000", 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001530039 2024-05-06T00:27:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301013+0000", 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8708000000000006e-05 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301197+0000", 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018319199999999999 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.304470+0000", 2024-05-06T00:27:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032733580000000001 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318928+0000", 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014458314 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318968+0000", 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9990000000000002e-05 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319010+0000", 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2160999999999998e-05 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3004 3.9 3:962ccfc1:::rbd_data.110cd5ddb45c.00000000000000d2:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.299411+0000", 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5703495260000002, 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019693321999999999, 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3004 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299411+0000", 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299411+0000", 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299412+0000", 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299414+0000", 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7480000000000002e-06 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299421+0000", 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2489999999999998e-06 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.299426+0000", 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8690000000000003e-06 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301531+0000", 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021053230000000001 2024-05-06T00:27:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301608+0000", 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6864000000000001e-05 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301784+0000", 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017621999999999999 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.304740+0000", 2024-05-06T00:27:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002955882 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.318964+0000", 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014224245 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319070+0000", 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105442 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319105+0000", 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5058999999999998e-05 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3025 3.9 3:97eed914:::rbd_data.110cd5ddb45c.00000000000000de:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.301506+0000", 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5682546820000001, 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017736934999999999, 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3025 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301506+0000", 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301506+0000", 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301507+0000", 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7899999999999998e-07 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301509+0000", 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301516+0000", 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301521+0000", 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2209999999999996e-06 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.302403+0000", 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00088221100000000004 2024-05-06T00:27:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.302517+0000", 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113596 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.319243+0000", 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016726320999999999 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:27.983 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:27.984 INFO:teuthology.orchestra.run.smithi186.stdout:1897 trunc from 0x7808600 to 0xc645800 2024-05-06T00:27:27.984 INFO:teuthology.orchestra.run.smithi186.stdout:1898 read 0xd0c00 thru 0xe055d (0xf95e bytes) 2024-05-06T00:27:27.984 INFO:teuthology.orchestra.run.smithi186.stdout:1900 trunc from 0xc645800 to 0x268a000 2024-05-06T00:27:27.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:27.985 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:27.979+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:28.021 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:27:28.029 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.1 to 1 (10000) 2024-05-06T00:27:28.053 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-05-06T00:27:28.097 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:28.098+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:28.133 INFO:teuthology.orchestra.run.smithi186.stdout:1901 write 0xa359800 thru 0xa35f0d3 (0x58d4 bytes) 2024-05-06T00:27:28.149 INFO:teuthology.orchestra.run.smithi186.stdout:1906 trunc from 0xa35f200 to 0xd09d200 2024-05-06T00:27:28.166 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:28.166 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:28.166 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:28.166 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:28.167 INFO:teuthology.orchestra.run.smithi186.stdout:1907 read 0xbd53c00 thru 0xbd54c0b (0x100c bytes) 2024-05-06T00:27:28.168 INFO:teuthology.orchestra.run.smithi186.stdout:1908 read 0xa06a000 thru 0xa076607 (0xc608 bytes) 2024-05-06T00:27:28.168 INFO:teuthology.orchestra.run.smithi186.stdout:1910 write 0x7d2c600 thru 0x7d3253f (0x5f40 bytes) 2024-05-06T00:27:28.168 INFO:teuthology.orchestra.run.smithi186.stdout:1912 punch from 0xc1abe00 to 0xc1b2400, (0x6600 bytes) 2024-05-06T00:27:28.169 INFO:teuthology.orchestra.run.smithi186.stdout:1915 punch from 0x6da7600 to 0x6daaa00, (0x3400 bytes) 2024-05-06T00:27:28.169 INFO:teuthology.orchestra.run.smithi186.stdout:1917 punch from 0x3381000 to 0x3381e00, (0xe00 bytes) 2024-05-06T00:27:28.169 INFO:teuthology.orchestra.run.smithi186.stdout:1919 read 0x9abec00 thru 0x9ac6341 (0x7742 bytes) 2024-05-06T00:27:28.169 INFO:teuthology.orchestra.run.smithi186.stdout:1921 read 0x4c59000 thru 0x4c5987d (0x87e bytes) 2024-05-06T00:27:28.170 INFO:teuthology.orchestra.run.smithi186.stdout:1933 write 0xa1cd600 thru 0xa1daab5 (0xd4b6 bytes) 2024-05-06T00:27:28.170 INFO:teuthology.orchestra.run.smithi186.stdout:1934 read 0x8f0000 thru 0x8fb96d (0xb96e bytes) 2024-05-06T00:27:28.171 INFO:teuthology.orchestra.run.smithi186.stdout:1936 write 0x7f76800 thru 0x7f83671 (0xce72 bytes) 2024-05-06T00:27:28.171 INFO:teuthology.orchestra.run.smithi186.stdout:1938 write 0x36f1600 thru 0x36f5996 (0x4397 bytes) 2024-05-06T00:27:28.171 INFO:teuthology.orchestra.run.smithi186.stdout:1939 read 0x55b7000 thru 0x55ba40d (0x340e bytes) 2024-05-06T00:27:28.171 INFO:teuthology.orchestra.run.smithi186.stdout:1943 read 0x1cfe00 thru 0x1db4d4 (0xb6d5 bytes) 2024-05-06T00:27:28.179 INFO:teuthology.orchestra.run.smithi186.stdout:1945 trunc from 0xd09d200 to 0xdfa0200 2024-05-06T00:27:28.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:28.186 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:28.186+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:28.206 INFO:teuthology.orchestra.run.smithi186.stdout:1948 read 0x5b96c00 thru 0x5ba1b92 (0xaf93 bytes) 2024-05-06T00:27:28.220 INFO:teuthology.orchestra.run.smithi186.stdout:1949 write 0xe057400 thru 0xe057c54 (0x855 bytes) 2024-05-06T00:27:28.220 INFO:teuthology.orchestra.run.smithi186.stdout:1950 read 0x2287600 thru 0x228c20c (0x4c0d bytes) 2024-05-06T00:27:28.222 INFO:teuthology.orchestra.run.smithi186.stdout:1951 read 0x9007000 thru 0x9007e8b (0xe8c bytes) 2024-05-06T00:27:28.242 INFO:teuthology.orchestra.run.smithi186.stdout:1952 trunc from 0xe057e00 to 0x3b8ae00 2024-05-06T00:27:28.269 INFO:teuthology.orchestra.run.smithi186.stdout:1953 read 0x2181c00 thru 0x21914d9 (0xf8da bytes) 2024-05-06T00:27:28.286 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:28.283+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:28.287 INFO:teuthology.orchestra.run.smithi186.stdout:1954 write 0x5bbfc00 thru 0x5bc32a3 (0x36a4 bytes) 2024-05-06T00:27:28.304 INFO:teuthology.orchestra.run.smithi186.stdout:1957 trunc from 0x5bc3400 to 0x1048e00 2024-05-06T00:27:28.307 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:28.307 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:28.307 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:28.307 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:28.307 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:28.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:28.387 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:28.386+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:28.413 INFO:teuthology.orchestra.run.smithi186.stdout:1959 write 0x45ff400 thru 0x4603ec6 (0x4ac7 bytes) 2024-05-06T00:27:28.415 INFO:teuthology.orchestra.run.smithi186.stdout:1960 write 0x593c00 thru 0x59f55b (0xb95c bytes) 2024-05-06T00:27:28.429 INFO:teuthology.orchestra.run.smithi186.stdout:1961 trunc from 0x4604000 to 0xa562000 2024-05-06T00:27:28.464 INFO:teuthology.orchestra.run.smithi186.stdout:1962 trunc from 0xa562000 to 0xe5c800 2024-05-06T00:27:28.488 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:28.483+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:28.492 INFO:teuthology.orchestra.run.smithi186.stdout:1967 read 0x5f8200 thru 0x5ffe5b (0x7c5c bytes) 2024-05-06T00:27:28.503 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:28.503 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:28.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:28.503 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.041730244, 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.504 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:27:28.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:27:28.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:27:28.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.040928255000001, 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.509 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:27:28.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:27:28.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.513 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.040841800999999, 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:27:28.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:27:28.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.518 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.040769208, 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.519 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:27:28.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.526 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.040709357000001, 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:27:28.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:27:28.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.040633674, 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2967 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 159744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.033009686, 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026274390000000002, 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2967 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467898+0000", 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.046e-06 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467917+0000", 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467937+0000", 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9613999999999999e-05 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467961+0000", 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.419e-05 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.493935+0000", 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025973922 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494053+0000", 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011866099999999999 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494171+0000", 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011806099999999999 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.032928101, 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.032873894, 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:27:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:28.549 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.032821745, 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:28.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3040 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.030966423000001, 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026298018999999999, 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3040 2024-05-06T00:27:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469941+0000", 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0979999999999999e-06 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469946+0000", 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0170000000000004e-06 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469963+0000", 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6541999999999998e-05 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469977+0000", 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4015999999999999e-05 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495954+0000", 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025976902999999999 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496101+0000", 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147529 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496238+0000", 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136914 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.03086403, 2024-05-06T00:27:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:27:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:27:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.03078769, 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2156499829999996, 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2155975420000003, 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:27:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:27:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:27:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:27:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2085775590000001, 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:27:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2085308479999997, 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2082007399999997, 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:27:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:27:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:27:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:27:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2081728180000004, 2024-05-06T00:27:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:27:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:27:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.1973059289999997, 2024-05-06T00:27:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:27:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:27:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:27:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:28.605 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:28.607 INFO:teuthology.orchestra.run.smithi186.stdout:1970 trunc from 0xe5c800 to 0xdece000 2024-05-06T00:27:28.607 INFO:teuthology.orchestra.run.smithi186.stdout:1972 read 0x6169400 thru 0x616d751 (0x4352 bytes) 2024-05-06T00:27:28.607 INFO:teuthology.orchestra.run.smithi186.stdout:1973 write 0x13db200 thru 0x13e771e (0xc51f bytes) 2024-05-06T00:27:28.607 INFO:teuthology.orchestra.run.smithi186.stdout:1975 write 0xebb3a00 thru 0xebbc9c1 (0x8fc2 bytes) 2024-05-06T00:27:28.607 INFO:teuthology.orchestra.run.smithi186.stdout:1976 read 0x512c400 thru 0x5136086 (0x9c87 bytes) 2024-05-06T00:27:28.607 INFO:teuthology.orchestra.run.smithi186.stdout:1980 punch from 0x68cd000 to 0x68db600, (0xe600 bytes) 2024-05-06T00:27:28.608 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:28.608 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:28.606+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:28.609 INFO:teuthology.orchestra.run.smithi186.stdout:1981 trunc from 0xebbca00 to 0x14c3800 2024-05-06T00:27:28.652 INFO:teuthology.orchestra.run.smithi186.stdout:1982 trunc from 0x14c3800 to 0x7a49e00 2024-05-06T00:27:28.668 INFO:teuthology.orchestra.run.smithi186.stdout:1983 clone 14 order 20 su 32768 sc 11 2024-05-06T00:27:28.694 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:28.745 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:28.746+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:28.745 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:28.746 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:28.746 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:28.746 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:28.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:28.808 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:28.807+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:28.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:28.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:28.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:28.869 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:28.890 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:28.890 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:28.910 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:28.907+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:28.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:28.921 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:28.921 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:28.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:28.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.2 to 1 (10000) 2024-05-06T00:27:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:27:29.026 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:29.026+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:29.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-05-06T00:27:29.112 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:29.107+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:29.122 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:29.122 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:29.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:29.224 INFO:teuthology.orchestra.run.smithi186.stdout:flattening image image_client.0-clone13 2024-05-06T00:27:29.225 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:29.226+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:29.314 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:29.314+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:29.317 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:29.414 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:29.414+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:29.485 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:29.486 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:29.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:29.515 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:29.511+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:29.616 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:29.615+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:29.666 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:29.666 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:29.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:29.729 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:29.730+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:29.803 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:29.803 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:29.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:29.825 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-05-06T00:27:29.861 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:29.862+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:29.919 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:29.918+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:29.982 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:29.982 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:29.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:30.020 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:30.015+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:30.036 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.3 to 1 (10000) 2024-05-06T00:27:30.061 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-05-06T00:27:30.121 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:30.119+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:30.167 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:30.167 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:30.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:30.221 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:30.222+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:30.322 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:30.322 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:30.322 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:30.322+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:30.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:30.367 INFO:teuthology.orchestra.run.smithi186.stdout:1984 read 0x3066a00 thru 0x306778e (0xd8f bytes) 2024-05-06T00:27:30.368 INFO:teuthology.orchestra.run.smithi186.stdout:1987 punch from 0x2e03000 to 0x2e0a600, (0x7600 bytes) 2024-05-06T00:27:30.389 INFO:teuthology.orchestra.run.smithi186.stdout:1988 read 0x22ce000 thru 0x22ceeaf (0xeb0 bytes) 2024-05-06T00:27:30.389 INFO:teuthology.orchestra.run.smithi186.stdout:1989 punch from 0xc62600 to 0xc66200, (0x3c00 bytes) 2024-05-06T00:27:30.424 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:30.419+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:30.459 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:30.459 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:30.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:30.486 INFO:teuthology.orchestra.run.smithi186.stdout:1990 write 0xe555a00 thru 0xe55d26f (0x7870 bytes) 2024-05-06T00:27:30.488 INFO:teuthology.orchestra.run.smithi186.stdout:1992 punch from 0x683c400 to 0x6847800, (0xb400 bytes) 2024-05-06T00:27:30.520 INFO:teuthology.orchestra.run.smithi186.stdout:1993 write 0xecd8000 thru 0xece3343 (0xb344 bytes) 2024-05-06T00:27:30.521 INFO:teuthology.orchestra.run.smithi186.stdout:1994 punch from 0x2702600 to 0x270b600, (0x9000 bytes) 2024-05-06T00:27:30.555 INFO:teuthology.orchestra.run.smithi186.stdout:1995 trunc from 0xece3400 to 0x6b24e00 2024-05-06T00:27:30.565 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:30.563+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:30.611 INFO:teuthology.orchestra.run.smithi186.stdout:1996 read 0x3c4f200 thru 0x3c571d5 (0x7fd6 bytes) 2024-05-06T00:27:30.613 INFO:teuthology.orchestra.run.smithi186.stdout:1997 read 0xf28400 thru 0xf36764 (0xe365 bytes) 2024-05-06T00:27:30.614 INFO:teuthology.orchestra.run.smithi186.stdout:1998 punch from 0x1f4ba00 to 0x1f53600, (0x7c00 bytes) 2024-05-06T00:27:30.616 INFO:teuthology.orchestra.run.smithi186.stdout:1999 write 0x3b9e000 thru 0x3ba9614 (0xb615 bytes) 2024-05-06T00:27:30.616 INFO:teuthology.orchestra.run.smithi186.stdout:2000 read 0x4558a00 thru 0x455f30e (0x690f bytes) 2024-05-06T00:27:30.617 INFO:teuthology.orchestra.run.smithi186.stdout:2002 read 0x4391000 thru 0x439ced4 (0xbed5 bytes) 2024-05-06T00:27:30.617 INFO:teuthology.orchestra.run.smithi186.stdout:2003 punch from 0x4cd7200 to 0x4ce7000, (0xfe00 bytes) 2024-05-06T00:27:30.618 INFO:teuthology.orchestra.run.smithi186.stdout:2005 read 0x2a99600 thru 0x2aa6de2 (0xd7e3 bytes) 2024-05-06T00:27:30.619 INFO:teuthology.orchestra.run.smithi186.stdout:2006 punch from 0x474d600 to 0x474de00, (0x800 bytes) 2024-05-06T00:27:30.619 INFO:teuthology.orchestra.run.smithi186.stdout:2008 read 0x2466600 thru 0x2468346 (0x1d47 bytes) 2024-05-06T00:27:30.620 INFO:teuthology.orchestra.run.smithi186.stdout:2010 write 0x1ba0400 thru 0x1ba0fed (0xbee bytes) 2024-05-06T00:27:30.620 INFO:teuthology.orchestra.run.smithi186.stdout:2011 read 0x31e5a00 thru 0x31f55a7 (0xfba8 bytes) 2024-05-06T00:27:30.620 INFO:teuthology.orchestra.run.smithi186.stdout:2012 punch from 0x3288200 to 0x328b800, (0x3600 bytes) 2024-05-06T00:27:30.626 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:30.623+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:30.626 INFO:teuthology.orchestra.run.smithi186.stdout:2015 trunc from 0x6b24e00 to 0x8e85800 2024-05-06T00:27:30.652 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:30.652 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:30.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:30.695 INFO:teuthology.orchestra.run.smithi186.stdout:2017 write 0xb8f1200 thru 0xb8f3413 (0x2214 bytes) 2024-05-06T00:27:30.697 INFO:teuthology.orchestra.run.smithi186.stdout:2019 write 0x3551800 thru 0x355221a (0xa1b bytes) 2024-05-06T00:27:30.697 INFO:teuthology.orchestra.run.smithi186.stdout:2021 write 0x469c800 thru 0x46a6e7f (0xa680 bytes) 2024-05-06T00:27:30.698 INFO:teuthology.orchestra.run.smithi186.stdout:2026 punch from 0x937c800 to 0x9388600, (0xbe00 bytes) 2024-05-06T00:27:30.698 INFO:teuthology.orchestra.run.smithi186.stdout:2028 punch from 0x680ba00 to 0x6811400, (0x5a00 bytes) 2024-05-06T00:27:30.699 INFO:teuthology.orchestra.run.smithi186.stdout:2029 write 0x1034000 thru 0x1035004 (0x1005 bytes) 2024-05-06T00:27:30.700 INFO:teuthology.orchestra.run.smithi186.stdout:2030 write 0x9da3400 thru 0x9db0cf8 (0xd8f9 bytes) 2024-05-06T00:27:30.701 INFO:teuthology.orchestra.run.smithi186.stdout:2031 clone 15 order 21 su 32768 sc 6 2024-05-06T00:27:30.754 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:30.751+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:30.828 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:30.823+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:30.834 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:30.834 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:30.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:30.929 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:30.930+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:30.951 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:31.013 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:31.013 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:31.027 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:31.029 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:31.030+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:31.044 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.4 to 1 (10000) 2024-05-06T00:27:31.059 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:27:31.070 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-05-06T00:27:31.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:31.115 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:31.133 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:31.133 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:31.133 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:31.134+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:31.157 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:31.157 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:31.163 INFO:teuthology.orchestra.run.smithi186.stdout:flattening image image_client.0-clone14 2024-05-06T00:27:31.165 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:31.165 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:31.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:31.197 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:31.198 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:31.231 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:31.230+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:31.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:31.234 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:31.254 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:31.306 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:31.306 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:31.306 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:31.306 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:31.320 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:31.365 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:31.363+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:31.433 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:31.431+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:31.440 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:31.441 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:31.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:31.541 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:31.539+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:31.627 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.166331817, 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.628 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:27:31.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:27:31.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.165529828, 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:31.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.637 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.165443374000001, 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:27:31.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:27:31.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.642 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.165370781, 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:27:31.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:27:31.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.16531093, 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:27:31.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.165235247, 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.652 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:31.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:27:31.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2967 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 159744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.157611258999999, 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026274390000000002, 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2967 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467898+0000", 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.046e-06 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467917+0000", 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467937+0000", 2024-05-06T00:27:31.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9613999999999999e-05 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467961+0000", 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.419e-05 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.493935+0000", 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025973922 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494053+0000", 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011866099999999999 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494171+0000", 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011806099999999999 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.661 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.157529673999999, 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.662 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:27:31.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.663 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:31.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:31.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:27:31.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.157475466999999, 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:27:31.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:27:31.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:27:31.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:27:31.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:27:31.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:27:31.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:27:31.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.157423317999999, 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.674 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.678 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3040 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.155567996, 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026298018999999999, 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3040 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469941+0000", 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0979999999999999e-06 2024-05-06T00:27:31.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469946+0000", 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0170000000000004e-06 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469963+0000", 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6541999999999998e-05 2024-05-06T00:27:31.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469977+0000", 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4015999999999999e-05 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495954+0000", 2024-05-06T00:27:31.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025976902999999999 2024-05-06T00:27:31.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496101+0000", 2024-05-06T00:27:31.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147529 2024-05-06T00:27:31.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496238+0000", 2024-05-06T00:27:31.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136914 2024-05-06T00:27:31.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.684 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.685 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.685 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.685 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:31.685 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.155465603, 2024-05-06T00:27:31.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:27:31.686 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.686 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.686 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.687 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:27:31.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.687 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:31.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:31.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:27:31.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:27:31.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:27:31.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:27:31.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:27:31.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:27:31.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:27:31.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:27:31.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:27:31.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:27:31.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:27:31.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:27:31.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:27:31.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:27:31.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.695 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.155389263, 2024-05-06T00:27:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:27:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:31.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:31.698 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:31.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.698 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:31.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:31.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:27:31.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:27:31.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:27:31.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:27:31.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:27:31.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:27:31.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:27:31.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:27:31.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:27:31.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:27:31.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:27:31.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:27:31.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:27:31.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:27:31.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.706 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.707 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.707 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:31.707 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:31.707 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3402515560000001, 2024-05-06T00:27:31.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:27:31.708 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.708 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.708 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:31.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:31.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:31.709 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:27:31.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.709 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:31.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:31.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:31.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:27:31.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:27:31.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:27:31.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:27:31.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:27:31.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:27:31.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:27:31.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:27:31.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:27:31.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:27:31.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:27:31.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:31.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:27:31.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:27:31.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:31.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:27:31.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:27:31.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:31.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:27:31.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3401991149999999, 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:27:31.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:27:31.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:31.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:27:31.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3331791319999997, 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:31.725 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:31.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:27:31.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3331324210000002, 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.731 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:27:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:31.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:27:31.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:27:31.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:31.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:27:31.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:27:31.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:31.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:27:31.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:27:31.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:27:31.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:27:31.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.739 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:31.739 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:31.739 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3328023130000002, 2024-05-06T00:27:31.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:27:31.739 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.739 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.739 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:31.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:31.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:31.740 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:27:31.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.740 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:31.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:31.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:31.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:27:31.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:27:31.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:27:31.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:27:31.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:27:31.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:27:31.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:27:31.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:27:31.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:27:31.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:27:31.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:27:31.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:31.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:27:31.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:27:31.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.749 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3327743910000001, 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.750 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:27:31.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:27:31.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:31.754 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3219075020000002, 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:31.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:27:31.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.759 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:31.759 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:31.759 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:31.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:31.761 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:31.758+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:31.780 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-05-06T00:27:31.861 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:31.862+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:31.895 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:31.895 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:31.895 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:31.895 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:31.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:31.997 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:31.998+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:32.063 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:32.059+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:32.064 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:27:32.065 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.5 to 1 (10000) 2024-05-06T00:27:32.089 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:32.089 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:32.090 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:32.090 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:32.090 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:32.093 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-05-06T00:27:32.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:27:32.107 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:32.193 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:32.194+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:32.265 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:32.266+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:32.285 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.480691542999999, 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.286 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:32.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:27:32.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.480675339999998, 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:27:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2527 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch ping cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.124581391, 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16552214100000001, 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2527 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160471+0000", 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.206e-06 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160477+0000", 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7440000000000001e-06 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160491+0000", 2024-05-06T00:27:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3739e-05 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160506+0000", 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.473e-05 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160712+0000", 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020643899999999999 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160736+0000", 2024-05-06T00:27:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3676000000000002e-05 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325517+0000", 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16478073700000001 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325685+0000", 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016879300000000001 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325992+0000", 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030607699999999998 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:317 3.1f 3:f834d203:::rbd_data.110cd5ddb45c.0000000000000015:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.142156+0000", 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1428949080000006, 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.045878791000000002, 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 317 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142156+0000", 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142156+0000", 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142156+0000", 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9999999999999997e-07 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142160+0000", 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7890000000000001e-06 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142876+0000", 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00071542299999999997 2024-05-06T00:27:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142889+0000", 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3213999999999999e-05 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.143181+0000", 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029239000000000001 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.143491+0000", 2024-05-06T00:27:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030928599999999998 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.188035+0000", 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.044544089000000002 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:322 3.12 3:4c45bc4e:::rbd_data.110cd5ddb45c.0000000000000018:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.142919+0000", 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1421314470000006, 2024-05-06T00:27:32.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.044706086999999999, 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 322 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142919+0000", 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142919+0000", 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142920+0000", 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9000000000000005e-07 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142925+0000", 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0119999999999996e-06 2024-05-06T00:27:32.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142937+0000", 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1471000000000001e-05 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142943+0000", 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6549999999999998e-06 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.143226+0000", 2024-05-06T00:27:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028248200000000002 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.143361+0000", 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013526400000000001 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187625+0000", 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.044264212999999997 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:642 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 4194304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:27.999341+0000", 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2857093649999998, 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.031487630000000003, 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 642 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:27.999341+0000", 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:27.999341+0000", 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:27.999343+0000", 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.237e-06 2024-05-06T00:27:32.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:27.999349+0000", 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.19e-06 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:27.999367+0000", 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8162999999999999e-05 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:27.999382+0000", 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5364000000000001e-05 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.014617+0000", 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015234231000000001 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.014854+0000", 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000237655 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.030829+0000", 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015974789999999999 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:834 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 4489216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.003483+0000", 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2815680980000002, 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.024557198999999998, 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 834 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.003483+0000", 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.003483+0000", 2024-05-06T00:27:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.003484+0000", 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9000000000000005e-07 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.003490+0000", 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.003508+0000", 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8467000000000001e-05 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.003525+0000", 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6263e-05 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.027584+0000", 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.024059176000000002 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.027804+0000", 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000220134 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.028040+0000", 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235999 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:893 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 2752512~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.004196+0000", 2024-05-06T00:27:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2808550629999997, 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.024707083000000001, 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 893 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004196+0000", 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004196+0000", 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004197+0000", 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4899999999999999e-06 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004206+0000", 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4370000000000007e-06 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004229+0000", 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3710000000000002e-05 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004246+0000", 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.626e-05 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.028606+0000", 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.024360357999999999 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.028740+0000", 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013388399999999999 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.028903+0000", 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016294399999999999 2024-05-06T00:27:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:894 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 3211264~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.004270+0000", 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2807803560000002, 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.025065859999999999, 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 894 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004270+0000", 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004270+0000", 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004272+0000", 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3069999999999999e-06 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004279+0000", 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2130000000000002e-06 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004300+0000", 2024-05-06T00:27:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0981000000000001e-05 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004315+0000", 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4769999999999999e-05 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.028969+0000", 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.024654723 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.029108+0000", 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013812399999999999 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.029336+0000", 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022874200000000001 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:895 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 3407872~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.004336+0000", 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2807148890000004, 2024-05-06T00:27:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.025563124, 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 895 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004336+0000", 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004336+0000", 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004337+0000", 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004343+0000", 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8599999999999998e-06 2024-05-06T00:27:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004361+0000", 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8300000000000001e-05 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004376+0000", 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.437e-05 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.029405+0000", 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.02502911 2024-05-06T00:27:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.029624+0000", 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021883499999999999 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.029899+0000", 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027544199999999998 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:896 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.004395+0000", 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.280656295, 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.026070645, 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 896 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004395+0000", 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.004395+0000", 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005027+0000", 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063205499999999996 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005038+0000", 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1199999999999999e-05 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005071+0000", 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2777000000000001e-05 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005089+0000", 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8482999999999999e-05 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.029970+0000", 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.024881240999999998 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.030186+0000", 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021554699999999999 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.030465+0000", 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027934200000000002 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:897 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 4063232~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005118+0000", 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.279932863, 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.025904480000000001, 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 897 2024-05-06T00:27:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005118+0000", 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005118+0000", 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005119+0000", 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.53e-06 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005156+0000", 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6467000000000002e-05 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005179+0000", 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3067000000000001e-05 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005215+0000", 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6004000000000003e-05 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.030538+0000", 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.025323116999999999 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.030746+0000", 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020749000000000001 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031022+0000", 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027680499999999998 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:898 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005237+0000", 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2798134780000003, 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.026217124000000001, 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 898 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005237+0000", 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005237+0000", 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005238+0000", 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0100000000000001e-06 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005244+0000", 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4060000000000004e-06 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005263+0000", 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.893e-05 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005278+0000", 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5097e-05 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031113+0000", 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.025835176000000001 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031333+0000", 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021964799999999999 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031454+0000", 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121857 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:956 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005298+0000", 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2797528480000002, 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.026393440000000001, 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 956 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005298+0000", 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005298+0000", 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005299+0000", 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.316e-06 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005308+0000", 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3140000000000004e-06 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005327+0000", 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8933e-05 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005341+0000", 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4076999999999999e-05 2024-05-06T00:27:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031490+0000", 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.026149136 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031576+0000", 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6225999999999997e-05 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031691+0000", 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000115438 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:957 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 2785280~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005360+0000", 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2796905110000001, 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.026785811, 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 957 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005360+0000", 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005360+0000", 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005361+0000", 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799999999999999e-06 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005367+0000", 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0039999999999996e-06 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005391+0000", 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3879999999999998e-05 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005407+0000", 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5855999999999999e-05 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031730+0000", 2024-05-06T00:27:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.026322478999999999 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.031912+0000", 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018192699999999999 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.032146+0000", 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023448499999999999 2024-05-06T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:958 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 3244032~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005426+0000", 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2796247970000003, 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027077291, 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 958 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005426+0000", 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005426+0000", 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005427+0000", 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1629999999999999e-06 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005433+0000", 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9399999999999999e-06 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005452+0000", 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8766999999999998e-05 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005467+0000", 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4707e-05 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.032228+0000", 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.026761017000000002 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.032353+0000", 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000125234 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.032503+0000", 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015046300000000001 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:959 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005486+0000", 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2795643029999999, 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027469968000000001, 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 959 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005486+0000", 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005486+0000", 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005488+0000", 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1000000000000001e-06 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005493+0000", 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5670000000000002e-06 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005512+0000", 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8442999999999999e-05 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005526+0000", 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4473e-05 2024-05-06T00:27:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.032542+0000", 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027015528 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.032647+0000", 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105764 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.032956+0000", 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030909299999999998 2024-05-06T00:27:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:960 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005544+0000", 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2795068499999998, 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027985956999999999, 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 960 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005544+0000", 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005544+0000", 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005545+0000", 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8700000000000004e-07 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005550+0000", 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3029999999999999e-06 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005569+0000", 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8666999999999999e-05 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005583+0000", 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4256999999999999e-05 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033017+0000", 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027434229000000001 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033119+0000", 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101821 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033530+0000", 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041069300000000001 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:961 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005602+0000", 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2794490559999998, 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.028159710000000001, 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 961 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005602+0000", 2024-05-06T00:27:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005602+0000", 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005603+0000", 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005609+0000", 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9000000000000003e-06 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005627+0000", 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8147000000000001e-05 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005641+0000", 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3869999999999999e-05 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033566+0000", 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027925695 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033657+0000", 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0457000000000005e-05 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033761+0000", 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104524 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:962 3.1b 3:ddca9df0:::rbd_data.110c5ae73073.0000000000000008:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005658+0000", 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2793923219999996, 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.028331619999999998, 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 962 2024-05-06T00:27:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005658+0000", 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005658+0000", 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005660+0000", 2024-05-06T00:27:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.203e-06 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005665+0000", 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.587e-06 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005683+0000", 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8073000000000001e-05 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005698+0000", 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.419e-05 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033796+0000", 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.028098172000000001 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033884+0000", 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8183999999999995e-05 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.033990+0000", 2024-05-06T00:27:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106211 2024-05-06T00:27:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.378 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:32.379 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:32.380 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:32.378+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:32.480 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:32.479+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:32.486 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:27:32.486 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:27:32.508 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-05-06T00:27:32.409001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":425483357,"num_objects":676,"num_object_clones":105,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":676,"num_whiteouts":439,"num_read":56118,"num_read_kb":170721,"num_write":5715,"num_write_kb":65607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5480,"ondisk_log_size":5480,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":246104,"kb_used_data":30168,"kb_used_omap":9,"kb_used_meta":211958,"kb_avail":565984936,"statfs":{"total":579820584960,"available":579568574464,"internally_reserved":0,"allocated":30892032,"data_stored":40931692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":217045664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000702"},"pg_stats":[{"pgid":"3.1f","version":"49'104","reported_seq":2559,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019673+0000","last_change":"2024-05-06T00:26:45.712771+0000","last_active":"2024-05-06T00:27:27.019673+0000","last_peered":"2024-05-06T00:27:27.019673+0000","last_clean":"2024-05-06T00:27:27.019673+0000","last_became_active":"2024-05-06T00:26:43.694494+0000","last_became_peered":"2024-05-06T00:26:43.694494+0000","last_unstale":"2024-05-06T00:27:27.019673+0000","last_undegraded":"2024-05-06T00:27:27.019673+0000","last_fullsized":"2024-05-06T00:27:27.019673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:57:34.217739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":2952758,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":14,"num_read":2408,"num_read_kb":12910,"num_write":93,"num_write_kb":616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"51'97","reported_seq":1110,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.294008+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:27:27.294008+0000","last_peered":"2024-05-06T00:27:27.294008+0000","last_clean":"2024-05-06T00:27:27.294008+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:27:27.294008+0000","last_undegraded":"2024-05-06T00:27:27.294008+0000","last_fullsized":"2024-05-06T00:27:27.294008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:05:19.418591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":40138240,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":12,"num_read":971,"num_read_kb":6516,"num_write":85,"num_write_kb":8819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"49'88","reported_seq":545,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013104+0000","last_change":"2024-05-06T00:26:45.733037+0000","last_active":"2024-05-06T00:27:27.013104+0000","last_peered":"2024-05-06T00:27:27.013104+0000","last_clean":"2024-05-06T00:27:27.013104+0000","last_became_active":"2024-05-06T00:26:43.711475+0000","last_became_peered":"2024-05-06T00:26:43.711475+0000","last_unstale":"2024-05-06T00:27:27.013104+0000","last_undegraded":"2024-05-06T00:27:27.013104+0000","last_fullsized":"2024-05-06T00:27:27.013104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:32:11.752924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":6266880,"num_objects":20,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":16,"num_read":382,"num_read_kb":1527,"num_write":72,"num_write_kb":415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"51'388","reported_seq":4685,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.409315+0000","last_change":"2024-05-06T00:26:45.732216+0000","last_active":"2024-05-06T00:27:27.409315+0000","last_peered":"2024-05-06T00:27:27.409315+0000","last_clean":"2024-05-06T00:27:27.409315+0000","last_became_active":"2024-05-06T00:26:43.715832+0000","last_became_peered":"2024-05-06T00:26:43.715832+0000","last_unstale":"2024-05-06T00:27:27.409315+0000","last_undegraded":"2024-05-06T00:27:27.409315+0000","last_fullsized":"2024-05-06T00:27:27.409315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:50:52.498801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":11161637,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":13,"num_read":6164,"num_read_kb":11957,"num_write":368,"num_write_kb":712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"51'327","reported_seq":2451,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.432115+0000","last_change":"2024-05-06T00:26:45.713563+0000","last_active":"2024-05-06T00:27:27.432115+0000","last_peered":"2024-05-06T00:27:27.432115+0000","last_clean":"2024-05-06T00:27:27.432115+0000","last_became_active":"2024-05-06T00:26:43.707253+0000","last_became_peered":"2024-05-06T00:26:43.707253+0000","last_unstale":"2024-05-06T00:27:27.432115+0000","last_undegraded":"2024-05-06T00:27:27.432115+0000","last_fullsized":"2024-05-06T00:27:27.432115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:30:15.071936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":37198922,"num_objects":35,"num_object_clones":9,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":19,"num_read":3496,"num_read_kb":11114,"num_write":331,"num_write_kb":3254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"51'117","reported_seq":2273,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.405469+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:27:27.405469+0000","last_peered":"2024-05-06T00:27:27.405469+0000","last_clean":"2024-05-06T00:27:27.405469+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:27:27.405469+0000","last_undegraded":"2024-05-06T00:27:27.405469+0000","last_fullsized":"2024-05-06T00:27:27.405469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:36:33.638100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":20819623,"num_objects":27,"num_object_clones":3,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":14,"num_read":2555,"num_read_kb":5268,"num_write":93,"num_write_kb":8777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"51'118","reported_seq":1043,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.009107+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:27:27.009107+0000","last_peered":"2024-05-06T00:27:27.009107+0000","last_clean":"2024-05-06T00:27:27.009107+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:27:27.009107+0000","last_undegraded":"2024-05-06T00:27:27.009107+0000","last_fullsized":"2024-05-06T00:27:27.009107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:49:03.188418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":11719680,"num_objects":12,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":7,"num_read":1546,"num_read_kb":1677,"num_write":88,"num_write_kb":658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"50'70","reported_seq":606,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.007511+0000","last_change":"2024-05-06T00:26:45.713666+0000","last_active":"2024-05-06T00:27:27.007511+0000","last_peered":"2024-05-06T00:27:27.007511+0000","last_clean":"2024-05-06T00:27:27.007511+0000","last_became_active":"2024-05-06T00:26:43.688284+0000","last_became_peered":"2024-05-06T00:26:43.688284+0000","last_unstale":"2024-05-06T00:27:27.007511+0000","last_undegraded":"2024-05-06T00:27:27.007511+0000","last_fullsized":"2024-05-06T00:27:27.007511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:32:54.255430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":131072,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":458,"num_read_kb":3507,"num_write":60,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"49'121","reported_seq":752,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019811+0000","last_change":"2024-05-06T00:26:45.713245+0000","last_active":"2024-05-06T00:27:27.019811+0000","last_peered":"2024-05-06T00:27:27.019811+0000","last_clean":"2024-05-06T00:27:27.019811+0000","last_became_active":"2024-05-06T00:26:43.698443+0000","last_became_peered":"2024-05-06T00:26:43.698443+0000","last_unstale":"2024-05-06T00:27:27.019811+0000","last_undegraded":"2024-05-06T00:27:27.019811+0000","last_fullsized":"2024-05-06T00:27:27.019811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:31:08.485984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":34119210,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":15,"num_read":632,"num_read_kb":3400,"num_write":106,"num_write_kb":2893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"51'83","reported_seq":745,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.293283+0000","last_change":"2024-05-06T00:26:45.733718+0000","last_active":"2024-05-06T00:27:27.293283+0000","last_peered":"2024-05-06T00:27:27.293283+0000","last_clean":"2024-05-06T00:27:27.293283+0000","last_became_active":"2024-05-06T00:26:43.703465+0000","last_became_peered":"2024-05-06T00:26:43.703465+0000","last_unstale":"2024-05-06T00:27:27.293283+0000","last_undegraded":"2024-05-06T00:27:27.293283+0000","last_fullsized":"2024-05-06T00:27:27.293283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:55:53.419922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":35580958,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":14,"num_read":641,"num_read_kb":1810,"num_write":76,"num_write_kb":495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"49'89","reported_seq":767,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015698+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:27:27.015698+0000","last_peered":"2024-05-06T00:27:27.015698+0000","last_clean":"2024-05-06T00:27:27.015698+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:27:27.015698+0000","last_undegraded":"2024-05-06T00:27:27.015698+0000","last_fullsized":"2024-05-06T00:27:27.015698+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:56:37.290498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983744,"num_objects":21,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":15,"num_read":655,"num_read_kb":2819,"num_write":78,"num_write_kb":11853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"49'88","reported_seq":771,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015784+0000","last_change":"2024-05-06T00:27:21.967099+0000","last_active":"2024-05-06T00:27:27.015784+0000","last_peered":"2024-05-06T00:27:27.015784+0000","last_clean":"2024-05-06T00:27:27.015784+0000","last_became_active":"2024-05-06T00:27:21.964399+0000","last_became_peered":"2024-05-06T00:27:21.964399+0000","last_unstale":"2024-05-06T00:27:27.015784+0000","last_undegraded":"2024-05-06T00:27:27.015784+0000","last_fullsized":"2024-05-06T00:27:27.015784+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:35:10.663632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":11730960,"num_objects":20,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":13,"num_read":616,"num_read_kb":3674,"num_write":67,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015903+0000","last_change":"2024-05-06T00:27:21.965066+0000","last_active":"2024-05-06T00:27:27.015903+0000","last_peered":"2024-05-06T00:27:27.015903+0000","last_clean":"2024-05-06T00:27:27.015903+0000","last_became_active":"2024-05-06T00:27:21.964272+0000","last_became_peered":"2024-05-06T00:27:21.964272+0000","last_unstale":"2024-05-06T00:27:27.015903+0000","last_undegraded":"2024-05-06T00:27:27.015903+0000","last_fullsized":"2024-05-06T00:27:27.015903+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:06.226485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"49'371","reported_seq":2043,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019776+0000","last_change":"2024-05-06T00:27:01.825846+0000","last_active":"2024-05-06T00:27:27.019776+0000","last_peered":"2024-05-06T00:27:27.019776+0000","last_clean":"2024-05-06T00:27:27.019776+0000","last_became_active":"2024-05-06T00:27:01.825382+0000","last_became_peered":"2024-05-06T00:27:01.825382+0000","last_unstale":"2024-05-06T00:27:27.019776+0000","last_undegraded":"2024-05-06T00:27:27.019776+0000","last_fullsized":"2024-05-06T00:27:27.019776+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:14:58.666831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":630866,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":11,"num_read":3993,"num_read_kb":3473,"num_write":433,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008689+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:27:27.008689+0000","last_peered":"2024-05-06T00:27:27.008689+0000","last_clean":"2024-05-06T00:27:27.008689+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:27:27.008689+0000","last_undegraded":"2024-05-06T00:27:27.008689+0000","last_fullsized":"2024-05-06T00:27:27.008689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"49'302","reported_seq":1261,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008083+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:27:27.008083+0000","last_peered":"2024-05-06T00:27:27.008083+0000","last_clean":"2024-05-06T00:27:27.008083+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:27:27.008083+0000","last_undegraded":"2024-05-06T00:27:27.008083+0000","last_fullsized":"2024-05-06T00:27:27.008083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:10:20.045677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":2088997,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":9,"num_read":2649,"num_read_kb":3608,"num_write":392,"num_write_kb":755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019485+0000","last_change":"2024-05-06T00:27:21.962747+0000","last_active":"2024-05-06T00:27:27.019485+0000","last_peered":"2024-05-06T00:27:27.019485+0000","last_clean":"2024-05-06T00:27:27.019485+0000","last_became_active":"2024-05-06T00:27:21.962380+0000","last_became_peered":"2024-05-06T00:27:21.962380+0000","last_unstale":"2024-05-06T00:27:27.019485+0000","last_undegraded":"2024-05-06T00:27:27.019485+0000","last_fullsized":"2024-05-06T00:27:27.019485+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:56.819600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"50'306","reported_seq":1356,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.009232+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:27:27.009232+0000","last_peered":"2024-05-06T00:27:27.009232+0000","last_clean":"2024-05-06T00:27:27.009232+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:27:27.009232+0000","last_undegraded":"2024-05-06T00:27:27.009232+0000","last_fullsized":"2024-05-06T00:27:27.009232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:11:53.473158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":20078149,"num_objects":31,"num_object_clones":6,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":20,"num_read":1856,"num_read_kb":9586,"num_write":396,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008749+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:27:27.008749+0000","last_peered":"2024-05-06T00:27:27.008749+0000","last_clean":"2024-05-06T00:27:27.008749+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:27:27.008749+0000","last_undegraded":"2024-05-06T00:27:27.008749+0000","last_fullsized":"2024-05-06T00:27:27.008749+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"49'86","reported_seq":556,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013273+0000","last_change":"2024-05-06T00:26:45.733467+0000","last_active":"2024-05-06T00:27:27.013273+0000","last_peered":"2024-05-06T00:27:27.013273+0000","last_clean":"2024-05-06T00:27:27.013273+0000","last_became_active":"2024-05-06T00:26:43.703416+0000","last_became_peered":"2024-05-06T00:26:43.703416+0000","last_unstale":"2024-05-06T00:27:27.013273+0000","last_undegraded":"2024-05-06T00:27:27.013273+0000","last_fullsized":"2024-05-06T00:27:27.013273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:08:15.699252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":745504,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":11,"num_read":411,"num_read_kb":2995,"num_write":74,"num_write_kb":472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"49'69","reported_seq":579,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015742+0000","last_change":"2024-05-06T00:26:45.720211+0000","last_active":"2024-05-06T00:27:27.015742+0000","last_peered":"2024-05-06T00:27:27.015742+0000","last_clean":"2024-05-06T00:27:27.015742+0000","last_became_active":"2024-05-06T00:26:43.705235+0000","last_became_peered":"2024-05-06T00:26:43.705235+0000","last_unstale":"2024-05-06T00:27:27.015742+0000","last_undegraded":"2024-05-06T00:27:27.015742+0000","last_fullsized":"2024-05-06T00:27:27.015742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:44:23.322220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":2341925,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":10,"num_read":439,"num_read_kb":3846,"num_write":59,"num_write_kb":417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013022+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:27:27.013022+0000","last_peered":"2024-05-06T00:27:27.013022+0000","last_clean":"2024-05-06T00:27:27.013022+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:27:27.013022+0000","last_undegraded":"2024-05-06T00:27:27.013022+0000","last_fullsized":"2024-05-06T00:27:27.013022+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"49'195","reported_seq":861,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008348+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:27:27.008348+0000","last_peered":"2024-05-06T00:27:27.008348+0000","last_clean":"2024-05-06T00:27:27.008348+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:27:27.008348+0000","last_undegraded":"2024-05-06T00:27:27.008348+0000","last_fullsized":"2024-05-06T00:27:27.008348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:40:17.004869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":5849192,"num_objects":20,"num_object_clones":1,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":12,"num_read":1120,"num_read_kb":3532,"num_write":252,"num_write_kb":878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.007234+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:27:27.007234+0000","last_peered":"2024-05-06T00:27:27.007234+0000","last_clean":"2024-05-06T00:27:27.007234+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:27:27.007234+0000","last_undegraded":"2024-05-06T00:27:27.007234+0000","last_fullsized":"2024-05-06T00:27:27.007234+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"49'84","reported_seq":615,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019470+0000","last_change":"2024-05-06T00:26:45.712444+0000","last_active":"2024-05-06T00:27:27.019470+0000","last_peered":"2024-05-06T00:27:27.019470+0000","last_clean":"2024-05-06T00:27:27.019470+0000","last_became_active":"2024-05-06T00:26:43.708328+0000","last_became_peered":"2024-05-06T00:26:43.708328+0000","last_unstale":"2024-05-06T00:27:27.019470+0000","last_undegraded":"2024-05-06T00:27:27.019470+0000","last_fullsized":"2024-05-06T00:27:27.019470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:32:44.852963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":22684672,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":11,"num_read":468,"num_read_kb":3916,"num_write":83,"num_write_kb":1081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008463+0000","last_change":"2024-05-06T00:27:21.957697+0000","last_active":"2024-05-06T00:27:27.008463+0000","last_peered":"2024-05-06T00:27:27.008463+0000","last_clean":"2024-05-06T00:27:27.008463+0000","last_became_active":"2024-05-06T00:27:21.957293+0000","last_became_peered":"2024-05-06T00:27:21.957293+0000","last_unstale":"2024-05-06T00:27:27.008463+0000","last_undegraded":"2024-05-06T00:27:27.008463+0000","last_fullsized":"2024-05-06T00:27:27.008463+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:36:37.494499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"51'195","reported_seq":2197,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.426222+0000","last_change":"2024-05-06T00:26:45.725541+0000","last_active":"2024-05-06T00:27:27.426222+0000","last_peered":"2024-05-06T00:27:27.426222+0000","last_clean":"2024-05-06T00:27:27.426222+0000","last_became_active":"2024-05-06T00:26:43.696938+0000","last_became_peered":"2024-05-06T00:26:43.696938+0000","last_unstale":"2024-05-06T00:27:27.426222+0000","last_undegraded":"2024-05-06T00:27:27.426222+0000","last_fullsized":"2024-05-06T00:27:27.426222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:22:43.523499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":8454202,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":10,"num_read":3041,"num_read_kb":4577,"num_write":133,"num_write_kb":801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013022+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:27:27.013022+0000","last_peered":"2024-05-06T00:27:27.013022+0000","last_clean":"2024-05-06T00:27:27.013022+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:27:27.013022+0000","last_undegraded":"2024-05-06T00:27:27.013022+0000","last_fullsized":"2024-05-06T00:27:27.013022+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":115,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019486+0000","last_change":"2024-05-06T00:27:21.961996+0000","last_active":"2024-05-06T00:27:27.019486+0000","last_peered":"2024-05-06T00:27:27.019486+0000","last_clean":"2024-05-06T00:27:27.019486+0000","last_became_active":"2024-05-06T00:27:21.961515+0000","last_became_peered":"2024-05-06T00:27:21.961515+0000","last_unstale":"2024-05-06T00:27:27.019486+0000","last_undegraded":"2024-05-06T00:27:27.019486+0000","last_fullsized":"2024-05-06T00:27:27.019486+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:36:39.984864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"49'248","reported_seq":1055,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.007999+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:27:27.007999+0000","last_peered":"2024-05-06T00:27:27.007999+0000","last_clean":"2024-05-06T00:27:27.007999+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:27:27.007999+0000","last_undegraded":"2024-05-06T00:27:27.007999+0000","last_fullsized":"2024-05-06T00:27:27.007999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:32:25.711846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":18699416,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":16,"num_read":1601,"num_read_kb":3290,"num_write":352,"num_write_kb":733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"50'89","reported_seq":555,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.112469+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:27:27.112469+0000","last_peered":"2024-05-06T00:27:27.112469+0000","last_clean":"2024-05-06T00:27:27.112469+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:27:27.112469+0000","last_undegraded":"2024-05-06T00:27:27.112469+0000","last_fullsized":"2024-05-06T00:27:27.112469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:04:53.169836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":8274510,"num_objects":20,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":15,"num_read":409,"num_read_kb":2177,"num_write":76,"num_write_kb":7213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"51'287","reported_seq":3749,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.344231+0000","last_change":"2024-05-06T00:27:21.968588+0000","last_active":"2024-05-06T00:27:27.344231+0000","last_peered":"2024-05-06T00:27:27.344231+0000","last_clean":"2024-05-06T00:27:27.344231+0000","last_became_active":"2024-05-06T00:27:21.967179+0000","last_became_peered":"2024-05-06T00:27:21.967179+0000","last_unstale":"2024-05-06T00:27:27.344231+0000","last_undegraded":"2024-05-06T00:27:27.344231+0000","last_fullsized":"2024-05-06T00:27:27.344231+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:29:16.072435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":17716911,"num_objects":37,"num_object_clones":8,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":19,"num_read":4913,"num_read_kb":12531,"num_write":192,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"51'87","reported_seq":482,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.290714+0000","last_change":"2024-05-06T00:26:45.725952+0000","last_active":"2024-05-06T00:27:27.290714+0000","last_peered":"2024-05-06T00:27:27.290714+0000","last_clean":"2024-05-06T00:27:27.290714+0000","last_became_active":"2024-05-06T00:26:43.686980+0000","last_became_peered":"2024-05-06T00:26:43.686980+0000","last_unstale":"2024-05-06T00:27:27.290714+0000","last_undegraded":"2024-05-06T00:27:27.290714+0000","last_fullsized":"2024-05-06T00:27:27.290714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:17:10.307562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":8528552,"num_objects":20,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":14,"num_read":364,"num_read_kb":2801,"num_write":71,"num_write_kb":3516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"50'105","reported_seq":899,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013259+0000","last_change":"2024-05-06T00:26:45.732742+0000","last_active":"2024-05-06T00:27:27.013259+0000","last_peered":"2024-05-06T00:27:27.013259+0000","last_clean":"2024-05-06T00:27:27.013259+0000","last_became_active":"2024-05-06T00:26:43.700115+0000","last_became_peered":"2024-05-06T00:26:43.700115+0000","last_unstale":"2024-05-06T00:27:27.013259+0000","last_undegraded":"2024-05-06T00:27:27.013259+0000","last_fullsized":"2024-05-06T00:27:27.013259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:18:33.170428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":3710976,"num_objects":21,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":16,"num_read":725,"num_read_kb":3508,"num_write":92,"num_write_kb":761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"51'534","reported_seq":1145,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.007977+0000","last_change":"2024-05-06T00:27:21.957565+0000","last_active":"2024-05-06T00:27:27.007977+0000","last_peered":"2024-05-06T00:27:27.007977+0000","last_clean":"2024-05-06T00:27:27.007977+0000","last_became_active":"2024-05-06T00:27:21.955822+0000","last_became_peered":"2024-05-06T00:27:21.955822+0000","last_unstale":"2024-05-06T00:27:27.007977+0000","last_undegraded":"2024-05-06T00:27:27.007977+0000","last_fullsized":"2024-05-06T00:27:27.007977+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:16:45.306325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":6614089,"num_objects":27,"num_object_clones":3,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":20,"num_read":2879,"num_read_kb":6078,"num_write":881,"num_write_kb":1347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"50'94","reported_seq":1508,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013315+0000","last_change":"2024-05-06T00:26:45.734276+0000","last_active":"2024-05-06T00:27:27.013315+0000","last_peered":"2024-05-06T00:27:27.013315+0000","last_clean":"2024-05-06T00:27:27.013315+0000","last_became_active":"2024-05-06T00:26:43.711491+0000","last_became_peered":"2024-05-06T00:26:43.711491+0000","last_unstale":"2024-05-06T00:27:27.013315+0000","last_undegraded":"2024-05-06T00:27:27.013315+0000","last_fullsized":"2024-05-06T00:27:27.013315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:21:48.469795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":9990144,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":15,"num_read":1890,"num_read_kb":5296,"num_write":60,"num_write_kb":331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"50'115","reported_seq":399,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013430+0000","last_change":"2024-05-06T00:26:45.734362+0000","last_active":"2024-05-06T00:27:27.013430+0000","last_peered":"2024-05-06T00:27:27.013430+0000","last_clean":"2024-05-06T00:27:27.013430+0000","last_became_active":"2024-05-06T00:26:43.703972+0000","last_became_peered":"2024-05-06T00:26:43.703972+0000","last_unstale":"2024-05-06T00:27:27.013430+0000","last_undegraded":"2024-05-06T00:27:27.013430+0000","last_fullsized":"2024-05-06T00:27:27.013430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:34:02.260918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":1040480,"num_objects":15,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":11,"num_read":451,"num_read_kb":2068,"num_write":136,"num_write_kb":462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"51'114","reported_seq":963,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013169+0000","last_change":"2024-05-06T00:26:45.732061+0000","last_active":"2024-05-06T00:27:27.013169+0000","last_peered":"2024-05-06T00:27:27.013169+0000","last_clean":"2024-05-06T00:27:27.013169+0000","last_became_active":"2024-05-06T00:26:43.699883+0000","last_became_peered":"2024-05-06T00:26:43.699883+0000","last_unstale":"2024-05-06T00:27:27.013169+0000","last_undegraded":"2024-05-06T00:27:27.013169+0000","last_fullsized":"2024-05-06T00:27:27.013169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:19:00.888789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":14244864,"num_objects":18,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":12,"num_read":1214,"num_read_kb":2941,"num_write":86,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"51'81","reported_seq":545,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015657+0000","last_change":"2024-05-06T00:27:21.968173+0000","last_active":"2024-05-06T00:27:27.015657+0000","last_peered":"2024-05-06T00:27:27.015657+0000","last_clean":"2024-05-06T00:27:27.015657+0000","last_became_active":"2024-05-06T00:27:21.965833+0000","last_became_peered":"2024-05-06T00:27:21.965833+0000","last_unstale":"2024-05-06T00:27:27.015657+0000","last_undegraded":"2024-05-06T00:27:27.015657+0000","last_fullsized":"2024-05-06T00:27:27.015657+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:39:51.732524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":15148032,"num_objects":22,"num_object_clones":2,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":19,"num_read":386,"num_read_kb":3367,"num_write":66,"num_write_kb":301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"50'132","reported_seq":1406,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019390+0000","last_change":"2024-05-06T00:26:45.713363+0000","last_active":"2024-05-06T00:27:27.019390+0000","last_peered":"2024-05-06T00:27:27.019390+0000","last_clean":"2024-05-06T00:27:27.019390+0000","last_became_active":"2024-05-06T00:26:43.692971+0000","last_became_peered":"2024-05-06T00:26:43.692971+0000","last_unstale":"2024-05-06T00:27:27.019390+0000","last_undegraded":"2024-05-06T00:27:27.019390+0000","last_fullsized":"2024-05-06T00:27:27.019390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:51:13.278704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":21017088,"num_objects":29,"num_object_clones":7,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":18,"num_read":1224,"num_read_kb":8333,"num_write":110,"num_write_kb":999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"51'224","reported_seq":5056,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.389594+0000","last_change":"2024-05-06T00:27:21.965729+0000","last_active":"2024-05-06T00:27:27.389594+0000","last_peered":"2024-05-06T00:27:27.389594+0000","last_clean":"2024-05-06T00:27:27.389594+0000","last_became_active":"2024-05-06T00:27:21.964410+0000","last_became_peered":"2024-05-06T00:27:21.964410+0000","last_unstale":"2024-05-06T00:27:27.389594+0000","last_undegraded":"2024-05-06T00:27:27.389594+0000","last_fullsized":"2024-05-06T00:27:27.389594+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:35:09.376083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":19230736,"num_objects":20,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":10,"num_read":5495,"num_read_kb":16564,"num_write":153,"num_write_kb":1274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":424892989,"num_objects":674,"num_object_clones":105,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":439,"num_read":56052,"num_read_kb":170666,"num_write":5614,"num_write_kb":63979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26255360,"data_stored":42480762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5408,"ondisk_log_size":5408,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640273,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37516,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334324,"statfs":{"total":96636764160,"available":96598347776,"internally_reserved":0,"allocated":2232320,"data_stored":2002218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640271,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41936,"kb_used_data":6600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329904,"statfs":{"total":96636764160,"available":96593821696,"internally_reserved":0,"allocated":6758400,"data_stored":10484365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640271,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42692,"kb_used_data":7352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329148,"statfs":{"total":96636764160,"available":96593047552,"internally_reserved":0,"allocated":7528448,"data_stored":11299807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640272,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44820,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327020,"statfs":{"total":96636764160,"available":96590868480,"internally_reserved":0,"allocated":5742592,"data_stored":5315604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640273,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":5688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":5824512,"data_stored":9467110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640271,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38080,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333760,"statfs":{"total":96636764160,"available":96597770240,"internally_reserved":0,"allocated":2805760,"data_stored":2362588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1650688,"data_stored":1605065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5246976,"data_stored":9239795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5165056,"data_stored":9121825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6356992,"data_stored":10482124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6180864,"data_stored":10257050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":1774903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:27:32.510 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:27:32.556 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:32.556 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:32.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:32.617 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:32.618+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:32.683 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:32.679+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:32.744 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:32.744 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:32.757 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:32.801 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:32.802+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:32.840 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:27:32.841 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:27:32.857 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-05-06T00:27:32.409001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":425483357,"num_objects":676,"num_object_clones":105,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":676,"num_whiteouts":439,"num_read":56118,"num_read_kb":170721,"num_write":5715,"num_write_kb":65607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5480,"ondisk_log_size":5480,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":246104,"kb_used_data":30168,"kb_used_omap":9,"kb_used_meta":211958,"kb_avail":565984936,"statfs":{"total":579820584960,"available":579568574464,"internally_reserved":0,"allocated":30892032,"data_stored":40931692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":217045664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000702"},"pg_stats":[{"pgid":"3.1f","version":"49'104","reported_seq":2559,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019673+0000","last_change":"2024-05-06T00:26:45.712771+0000","last_active":"2024-05-06T00:27:27.019673+0000","last_peered":"2024-05-06T00:27:27.019673+0000","last_clean":"2024-05-06T00:27:27.019673+0000","last_became_active":"2024-05-06T00:26:43.694494+0000","last_became_peered":"2024-05-06T00:26:43.694494+0000","last_unstale":"2024-05-06T00:27:27.019673+0000","last_undegraded":"2024-05-06T00:27:27.019673+0000","last_fullsized":"2024-05-06T00:27:27.019673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:57:34.217739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":2952758,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":14,"num_read":2408,"num_read_kb":12910,"num_write":93,"num_write_kb":616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"51'97","reported_seq":1110,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.294008+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:27:27.294008+0000","last_peered":"2024-05-06T00:27:27.294008+0000","last_clean":"2024-05-06T00:27:27.294008+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:27:27.294008+0000","last_undegraded":"2024-05-06T00:27:27.294008+0000","last_fullsized":"2024-05-06T00:27:27.294008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:05:19.418591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":40138240,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":12,"num_read":971,"num_read_kb":6516,"num_write":85,"num_write_kb":8819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"49'88","reported_seq":545,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013104+0000","last_change":"2024-05-06T00:26:45.733037+0000","last_active":"2024-05-06T00:27:27.013104+0000","last_peered":"2024-05-06T00:27:27.013104+0000","last_clean":"2024-05-06T00:27:27.013104+0000","last_became_active":"2024-05-06T00:26:43.711475+0000","last_became_peered":"2024-05-06T00:26:43.711475+0000","last_unstale":"2024-05-06T00:27:27.013104+0000","last_undegraded":"2024-05-06T00:27:27.013104+0000","last_fullsized":"2024-05-06T00:27:27.013104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:32:11.752924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":6266880,"num_objects":20,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":16,"num_read":382,"num_read_kb":1527,"num_write":72,"num_write_kb":415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"51'388","reported_seq":4685,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.409315+0000","last_change":"2024-05-06T00:26:45.732216+0000","last_active":"2024-05-06T00:27:27.409315+0000","last_peered":"2024-05-06T00:27:27.409315+0000","last_clean":"2024-05-06T00:27:27.409315+0000","last_became_active":"2024-05-06T00:26:43.715832+0000","last_became_peered":"2024-05-06T00:26:43.715832+0000","last_unstale":"2024-05-06T00:27:27.409315+0000","last_undegraded":"2024-05-06T00:27:27.409315+0000","last_fullsized":"2024-05-06T00:27:27.409315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:50:52.498801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":11161637,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":13,"num_read":6164,"num_read_kb":11957,"num_write":368,"num_write_kb":712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"51'327","reported_seq":2451,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.432115+0000","last_change":"2024-05-06T00:26:45.713563+0000","last_active":"2024-05-06T00:27:27.432115+0000","last_peered":"2024-05-06T00:27:27.432115+0000","last_clean":"2024-05-06T00:27:27.432115+0000","last_became_active":"2024-05-06T00:26:43.707253+0000","last_became_peered":"2024-05-06T00:26:43.707253+0000","last_unstale":"2024-05-06T00:27:27.432115+0000","last_undegraded":"2024-05-06T00:27:27.432115+0000","last_fullsized":"2024-05-06T00:27:27.432115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:30:15.071936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":37198922,"num_objects":35,"num_object_clones":9,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":19,"num_read":3496,"num_read_kb":11114,"num_write":331,"num_write_kb":3254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"51'117","reported_seq":2273,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.405469+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:27:27.405469+0000","last_peered":"2024-05-06T00:27:27.405469+0000","last_clean":"2024-05-06T00:27:27.405469+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:27:27.405469+0000","last_undegraded":"2024-05-06T00:27:27.405469+0000","last_fullsized":"2024-05-06T00:27:27.405469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:36:33.638100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":20819623,"num_objects":27,"num_object_clones":3,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":14,"num_read":2555,"num_read_kb":5268,"num_write":93,"num_write_kb":8777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"51'118","reported_seq":1043,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.009107+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:27:27.009107+0000","last_peered":"2024-05-06T00:27:27.009107+0000","last_clean":"2024-05-06T00:27:27.009107+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:27:27.009107+0000","last_undegraded":"2024-05-06T00:27:27.009107+0000","last_fullsized":"2024-05-06T00:27:27.009107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:49:03.188418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":11719680,"num_objects":12,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":7,"num_read":1546,"num_read_kb":1677,"num_write":88,"num_write_kb":658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"50'70","reported_seq":606,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.007511+0000","last_change":"2024-05-06T00:26:45.713666+0000","last_active":"2024-05-06T00:27:27.007511+0000","last_peered":"2024-05-06T00:27:27.007511+0000","last_clean":"2024-05-06T00:27:27.007511+0000","last_became_active":"2024-05-06T00:26:43.688284+0000","last_became_peered":"2024-05-06T00:26:43.688284+0000","last_unstale":"2024-05-06T00:27:27.007511+0000","last_undegraded":"2024-05-06T00:27:27.007511+0000","last_fullsized":"2024-05-06T00:27:27.007511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:32:54.255430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":131072,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":458,"num_read_kb":3507,"num_write":60,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"49'121","reported_seq":752,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019811+0000","last_change":"2024-05-06T00:26:45.713245+0000","last_active":"2024-05-06T00:27:27.019811+0000","last_peered":"2024-05-06T00:27:27.019811+0000","last_clean":"2024-05-06T00:27:27.019811+0000","last_became_active":"2024-05-06T00:26:43.698443+0000","last_became_peered":"2024-05-06T00:26:43.698443+0000","last_unstale":"2024-05-06T00:27:27.019811+0000","last_undegraded":"2024-05-06T00:27:27.019811+0000","last_fullsized":"2024-05-06T00:27:27.019811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:31:08.485984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":34119210,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":15,"num_read":632,"num_read_kb":3400,"num_write":106,"num_write_kb":2893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"51'83","reported_seq":745,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.293283+0000","last_change":"2024-05-06T00:26:45.733718+0000","last_active":"2024-05-06T00:27:27.293283+0000","last_peered":"2024-05-06T00:27:27.293283+0000","last_clean":"2024-05-06T00:27:27.293283+0000","last_became_active":"2024-05-06T00:26:43.703465+0000","last_became_peered":"2024-05-06T00:26:43.703465+0000","last_unstale":"2024-05-06T00:27:27.293283+0000","last_undegraded":"2024-05-06T00:27:27.293283+0000","last_fullsized":"2024-05-06T00:27:27.293283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:55:53.419922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":35580958,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":14,"num_read":641,"num_read_kb":1810,"num_write":76,"num_write_kb":495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"49'89","reported_seq":767,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015698+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:27:27.015698+0000","last_peered":"2024-05-06T00:27:27.015698+0000","last_clean":"2024-05-06T00:27:27.015698+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:27:27.015698+0000","last_undegraded":"2024-05-06T00:27:27.015698+0000","last_fullsized":"2024-05-06T00:27:27.015698+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:56:37.290498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983744,"num_objects":21,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":15,"num_read":655,"num_read_kb":2819,"num_write":78,"num_write_kb":11853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"49'88","reported_seq":771,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015784+0000","last_change":"2024-05-06T00:27:21.967099+0000","last_active":"2024-05-06T00:27:27.015784+0000","last_peered":"2024-05-06T00:27:27.015784+0000","last_clean":"2024-05-06T00:27:27.015784+0000","last_became_active":"2024-05-06T00:27:21.964399+0000","last_became_peered":"2024-05-06T00:27:21.964399+0000","last_unstale":"2024-05-06T00:27:27.015784+0000","last_undegraded":"2024-05-06T00:27:27.015784+0000","last_fullsized":"2024-05-06T00:27:27.015784+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:35:10.663632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":11730960,"num_objects":20,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":13,"num_read":616,"num_read_kb":3674,"num_write":67,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015903+0000","last_change":"2024-05-06T00:27:21.965066+0000","last_active":"2024-05-06T00:27:27.015903+0000","last_peered":"2024-05-06T00:27:27.015903+0000","last_clean":"2024-05-06T00:27:27.015903+0000","last_became_active":"2024-05-06T00:27:21.964272+0000","last_became_peered":"2024-05-06T00:27:21.964272+0000","last_unstale":"2024-05-06T00:27:27.015903+0000","last_undegraded":"2024-05-06T00:27:27.015903+0000","last_fullsized":"2024-05-06T00:27:27.015903+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:06.226485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"49'371","reported_seq":2043,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019776+0000","last_change":"2024-05-06T00:27:01.825846+0000","last_active":"2024-05-06T00:27:27.019776+0000","last_peered":"2024-05-06T00:27:27.019776+0000","last_clean":"2024-05-06T00:27:27.019776+0000","last_became_active":"2024-05-06T00:27:01.825382+0000","last_became_peered":"2024-05-06T00:27:01.825382+0000","last_unstale":"2024-05-06T00:27:27.019776+0000","last_undegraded":"2024-05-06T00:27:27.019776+0000","last_fullsized":"2024-05-06T00:27:27.019776+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:14:58.666831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":630866,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":11,"num_read":3993,"num_read_kb":3473,"num_write":433,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008689+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:27:27.008689+0000","last_peered":"2024-05-06T00:27:27.008689+0000","last_clean":"2024-05-06T00:27:27.008689+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:27:27.008689+0000","last_undegraded":"2024-05-06T00:27:27.008689+0000","last_fullsized":"2024-05-06T00:27:27.008689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"49'302","reported_seq":1261,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008083+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:27:27.008083+0000","last_peered":"2024-05-06T00:27:27.008083+0000","last_clean":"2024-05-06T00:27:27.008083+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:27:27.008083+0000","last_undegraded":"2024-05-06T00:27:27.008083+0000","last_fullsized":"2024-05-06T00:27:27.008083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:10:20.045677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":2088997,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":9,"num_read":2649,"num_read_kb":3608,"num_write":392,"num_write_kb":755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019485+0000","last_change":"2024-05-06T00:27:21.962747+0000","last_active":"2024-05-06T00:27:27.019485+0000","last_peered":"2024-05-06T00:27:27.019485+0000","last_clean":"2024-05-06T00:27:27.019485+0000","last_became_active":"2024-05-06T00:27:21.962380+0000","last_became_peered":"2024-05-06T00:27:21.962380+0000","last_unstale":"2024-05-06T00:27:27.019485+0000","last_undegraded":"2024-05-06T00:27:27.019485+0000","last_fullsized":"2024-05-06T00:27:27.019485+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:56.819600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"50'306","reported_seq":1356,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.009232+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:27:27.009232+0000","last_peered":"2024-05-06T00:27:27.009232+0000","last_clean":"2024-05-06T00:27:27.009232+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:27:27.009232+0000","last_undegraded":"2024-05-06T00:27:27.009232+0000","last_fullsized":"2024-05-06T00:27:27.009232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:11:53.473158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":20078149,"num_objects":31,"num_object_clones":6,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":20,"num_read":1856,"num_read_kb":9586,"num_write":396,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008749+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:27:27.008749+0000","last_peered":"2024-05-06T00:27:27.008749+0000","last_clean":"2024-05-06T00:27:27.008749+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:27:27.008749+0000","last_undegraded":"2024-05-06T00:27:27.008749+0000","last_fullsized":"2024-05-06T00:27:27.008749+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"49'86","reported_seq":556,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013273+0000","last_change":"2024-05-06T00:26:45.733467+0000","last_active":"2024-05-06T00:27:27.013273+0000","last_peered":"2024-05-06T00:27:27.013273+0000","last_clean":"2024-05-06T00:27:27.013273+0000","last_became_active":"2024-05-06T00:26:43.703416+0000","last_became_peered":"2024-05-06T00:26:43.703416+0000","last_unstale":"2024-05-06T00:27:27.013273+0000","last_undegraded":"2024-05-06T00:27:27.013273+0000","last_fullsized":"2024-05-06T00:27:27.013273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:08:15.699252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":745504,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":11,"num_read":411,"num_read_kb":2995,"num_write":74,"num_write_kb":472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"49'69","reported_seq":579,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015742+0000","last_change":"2024-05-06T00:26:45.720211+0000","last_active":"2024-05-06T00:27:27.015742+0000","last_peered":"2024-05-06T00:27:27.015742+0000","last_clean":"2024-05-06T00:27:27.015742+0000","last_became_active":"2024-05-06T00:26:43.705235+0000","last_became_peered":"2024-05-06T00:26:43.705235+0000","last_unstale":"2024-05-06T00:27:27.015742+0000","last_undegraded":"2024-05-06T00:27:27.015742+0000","last_fullsized":"2024-05-06T00:27:27.015742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:44:23.322220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":2341925,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":10,"num_read":439,"num_read_kb":3846,"num_write":59,"num_write_kb":417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013022+0000","last_change":"2024-05-06T00:26:35.653019+0000","last_active":"2024-05-06T00:27:27.013022+0000","last_peered":"2024-05-06T00:27:27.013022+0000","last_clean":"2024-05-06T00:27:27.013022+0000","last_became_active":"2024-05-06T00:26:35.652245+0000","last_became_peered":"2024-05-06T00:26:35.652245+0000","last_unstale":"2024-05-06T00:27:27.013022+0000","last_undegraded":"2024-05-06T00:27:27.013022+0000","last_fullsized":"2024-05-06T00:27:27.013022+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:19:12.248096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"49'195","reported_seq":861,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008348+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:27:27.008348+0000","last_peered":"2024-05-06T00:27:27.008348+0000","last_clean":"2024-05-06T00:27:27.008348+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:27:27.008348+0000","last_undegraded":"2024-05-06T00:27:27.008348+0000","last_fullsized":"2024-05-06T00:27:27.008348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:40:17.004869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":5849192,"num_objects":20,"num_object_clones":1,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":12,"num_read":1120,"num_read_kb":3532,"num_write":252,"num_write_kb":878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.007234+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:27:27.007234+0000","last_peered":"2024-05-06T00:27:27.007234+0000","last_clean":"2024-05-06T00:27:27.007234+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:27:27.007234+0000","last_undegraded":"2024-05-06T00:27:27.007234+0000","last_fullsized":"2024-05-06T00:27:27.007234+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"49'84","reported_seq":615,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019470+0000","last_change":"2024-05-06T00:26:45.712444+0000","last_active":"2024-05-06T00:27:27.019470+0000","last_peered":"2024-05-06T00:27:27.019470+0000","last_clean":"2024-05-06T00:27:27.019470+0000","last_became_active":"2024-05-06T00:26:43.708328+0000","last_became_peered":"2024-05-06T00:26:43.708328+0000","last_unstale":"2024-05-06T00:27:27.019470+0000","last_undegraded":"2024-05-06T00:27:27.019470+0000","last_fullsized":"2024-05-06T00:27:27.019470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:32:44.852963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":22684672,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":11,"num_read":468,"num_read_kb":3916,"num_write":83,"num_write_kb":1081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.008463+0000","last_change":"2024-05-06T00:27:21.957697+0000","last_active":"2024-05-06T00:27:27.008463+0000","last_peered":"2024-05-06T00:27:27.008463+0000","last_clean":"2024-05-06T00:27:27.008463+0000","last_became_active":"2024-05-06T00:27:21.957293+0000","last_became_peered":"2024-05-06T00:27:21.957293+0000","last_unstale":"2024-05-06T00:27:27.008463+0000","last_undegraded":"2024-05-06T00:27:27.008463+0000","last_fullsized":"2024-05-06T00:27:27.008463+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:36:37.494499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"51'195","reported_seq":2197,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.426222+0000","last_change":"2024-05-06T00:26:45.725541+0000","last_active":"2024-05-06T00:27:27.426222+0000","last_peered":"2024-05-06T00:27:27.426222+0000","last_clean":"2024-05-06T00:27:27.426222+0000","last_became_active":"2024-05-06T00:26:43.696938+0000","last_became_peered":"2024-05-06T00:26:43.696938+0000","last_unstale":"2024-05-06T00:27:27.426222+0000","last_undegraded":"2024-05-06T00:27:27.426222+0000","last_fullsized":"2024-05-06T00:27:27.426222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:22:43.523499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":8454202,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":10,"num_read":3041,"num_read_kb":4577,"num_write":133,"num_write_kb":801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":80,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013022+0000","last_change":"2024-05-06T00:26:35.649355+0000","last_active":"2024-05-06T00:27:27.013022+0000","last_peered":"2024-05-06T00:27:27.013022+0000","last_clean":"2024-05-06T00:27:27.013022+0000","last_became_active":"2024-05-06T00:26:35.648455+0000","last_became_peered":"2024-05-06T00:26:35.648455+0000","last_unstale":"2024-05-06T00:27:27.013022+0000","last_undegraded":"2024-05-06T00:27:27.013022+0000","last_fullsized":"2024-05-06T00:27:27.013022+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:46:54.237724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":115,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019486+0000","last_change":"2024-05-06T00:27:21.961996+0000","last_active":"2024-05-06T00:27:27.019486+0000","last_peered":"2024-05-06T00:27:27.019486+0000","last_clean":"2024-05-06T00:27:27.019486+0000","last_became_active":"2024-05-06T00:27:21.961515+0000","last_became_peered":"2024-05-06T00:27:21.961515+0000","last_unstale":"2024-05-06T00:27:27.019486+0000","last_undegraded":"2024-05-06T00:27:27.019486+0000","last_fullsized":"2024-05-06T00:27:27.019486+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:36:39.984864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"49'248","reported_seq":1055,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.007999+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:27:27.007999+0000","last_peered":"2024-05-06T00:27:27.007999+0000","last_clean":"2024-05-06T00:27:27.007999+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:27:27.007999+0000","last_undegraded":"2024-05-06T00:27:27.007999+0000","last_fullsized":"2024-05-06T00:27:27.007999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:32:25.711846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":18699416,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":16,"num_read":1601,"num_read_kb":3290,"num_write":352,"num_write_kb":733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"50'89","reported_seq":555,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.112469+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:27:27.112469+0000","last_peered":"2024-05-06T00:27:27.112469+0000","last_clean":"2024-05-06T00:27:27.112469+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:27:27.112469+0000","last_undegraded":"2024-05-06T00:27:27.112469+0000","last_fullsized":"2024-05-06T00:27:27.112469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:04:53.169836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":8274510,"num_objects":20,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":15,"num_read":409,"num_read_kb":2177,"num_write":76,"num_write_kb":7213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"51'287","reported_seq":3749,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.344231+0000","last_change":"2024-05-06T00:27:21.968588+0000","last_active":"2024-05-06T00:27:27.344231+0000","last_peered":"2024-05-06T00:27:27.344231+0000","last_clean":"2024-05-06T00:27:27.344231+0000","last_became_active":"2024-05-06T00:27:21.967179+0000","last_became_peered":"2024-05-06T00:27:21.967179+0000","last_unstale":"2024-05-06T00:27:27.344231+0000","last_undegraded":"2024-05-06T00:27:27.344231+0000","last_fullsized":"2024-05-06T00:27:27.344231+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:29:16.072435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":17716911,"num_objects":37,"num_object_clones":8,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":19,"num_read":4913,"num_read_kb":12531,"num_write":192,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"51'87","reported_seq":482,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.290714+0000","last_change":"2024-05-06T00:26:45.725952+0000","last_active":"2024-05-06T00:27:27.290714+0000","last_peered":"2024-05-06T00:27:27.290714+0000","last_clean":"2024-05-06T00:27:27.290714+0000","last_became_active":"2024-05-06T00:26:43.686980+0000","last_became_peered":"2024-05-06T00:26:43.686980+0000","last_unstale":"2024-05-06T00:27:27.290714+0000","last_undegraded":"2024-05-06T00:27:27.290714+0000","last_fullsized":"2024-05-06T00:27:27.290714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:17:10.307562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":8528552,"num_objects":20,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":14,"num_read":364,"num_read_kb":2801,"num_write":71,"num_write_kb":3516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"50'105","reported_seq":899,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013259+0000","last_change":"2024-05-06T00:26:45.732742+0000","last_active":"2024-05-06T00:27:27.013259+0000","last_peered":"2024-05-06T00:27:27.013259+0000","last_clean":"2024-05-06T00:27:27.013259+0000","last_became_active":"2024-05-06T00:26:43.700115+0000","last_became_peered":"2024-05-06T00:26:43.700115+0000","last_unstale":"2024-05-06T00:27:27.013259+0000","last_undegraded":"2024-05-06T00:27:27.013259+0000","last_fullsized":"2024-05-06T00:27:27.013259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:18:33.170428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":3710976,"num_objects":21,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":16,"num_read":725,"num_read_kb":3508,"num_write":92,"num_write_kb":761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"51'534","reported_seq":1145,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.007977+0000","last_change":"2024-05-06T00:27:21.957565+0000","last_active":"2024-05-06T00:27:27.007977+0000","last_peered":"2024-05-06T00:27:27.007977+0000","last_clean":"2024-05-06T00:27:27.007977+0000","last_became_active":"2024-05-06T00:27:21.955822+0000","last_became_peered":"2024-05-06T00:27:21.955822+0000","last_unstale":"2024-05-06T00:27:27.007977+0000","last_undegraded":"2024-05-06T00:27:27.007977+0000","last_fullsized":"2024-05-06T00:27:27.007977+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:16:45.306325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":6614089,"num_objects":27,"num_object_clones":3,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":20,"num_read":2879,"num_read_kb":6078,"num_write":881,"num_write_kb":1347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"50'94","reported_seq":1508,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013315+0000","last_change":"2024-05-06T00:26:45.734276+0000","last_active":"2024-05-06T00:27:27.013315+0000","last_peered":"2024-05-06T00:27:27.013315+0000","last_clean":"2024-05-06T00:27:27.013315+0000","last_became_active":"2024-05-06T00:26:43.711491+0000","last_became_peered":"2024-05-06T00:26:43.711491+0000","last_unstale":"2024-05-06T00:27:27.013315+0000","last_undegraded":"2024-05-06T00:27:27.013315+0000","last_fullsized":"2024-05-06T00:27:27.013315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:21:48.469795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":9990144,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":15,"num_read":1890,"num_read_kb":5296,"num_write":60,"num_write_kb":331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"50'115","reported_seq":399,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013430+0000","last_change":"2024-05-06T00:26:45.734362+0000","last_active":"2024-05-06T00:27:27.013430+0000","last_peered":"2024-05-06T00:27:27.013430+0000","last_clean":"2024-05-06T00:27:27.013430+0000","last_became_active":"2024-05-06T00:26:43.703972+0000","last_became_peered":"2024-05-06T00:26:43.703972+0000","last_unstale":"2024-05-06T00:27:27.013430+0000","last_undegraded":"2024-05-06T00:27:27.013430+0000","last_fullsized":"2024-05-06T00:27:27.013430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:34:02.260918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":1040480,"num_objects":15,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":11,"num_read":451,"num_read_kb":2068,"num_write":136,"num_write_kb":462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"51'114","reported_seq":963,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.013169+0000","last_change":"2024-05-06T00:26:45.732061+0000","last_active":"2024-05-06T00:27:27.013169+0000","last_peered":"2024-05-06T00:27:27.013169+0000","last_clean":"2024-05-06T00:27:27.013169+0000","last_became_active":"2024-05-06T00:26:43.699883+0000","last_became_peered":"2024-05-06T00:26:43.699883+0000","last_unstale":"2024-05-06T00:27:27.013169+0000","last_undegraded":"2024-05-06T00:27:27.013169+0000","last_fullsized":"2024-05-06T00:27:27.013169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:19:00.888789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":14244864,"num_objects":18,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":12,"num_read":1214,"num_read_kb":2941,"num_write":86,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"51'81","reported_seq":545,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.015657+0000","last_change":"2024-05-06T00:27:21.968173+0000","last_active":"2024-05-06T00:27:27.015657+0000","last_peered":"2024-05-06T00:27:27.015657+0000","last_clean":"2024-05-06T00:27:27.015657+0000","last_became_active":"2024-05-06T00:27:21.965833+0000","last_became_peered":"2024-05-06T00:27:21.965833+0000","last_unstale":"2024-05-06T00:27:27.015657+0000","last_undegraded":"2024-05-06T00:27:27.015657+0000","last_fullsized":"2024-05-06T00:27:27.015657+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:39:51.732524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":15148032,"num_objects":22,"num_object_clones":2,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":19,"num_read":386,"num_read_kb":3367,"num_write":66,"num_write_kb":301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"50'132","reported_seq":1406,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.019390+0000","last_change":"2024-05-06T00:26:45.713363+0000","last_active":"2024-05-06T00:27:27.019390+0000","last_peered":"2024-05-06T00:27:27.019390+0000","last_clean":"2024-05-06T00:27:27.019390+0000","last_became_active":"2024-05-06T00:26:43.692971+0000","last_became_peered":"2024-05-06T00:26:43.692971+0000","last_unstale":"2024-05-06T00:27:27.019390+0000","last_undegraded":"2024-05-06T00:27:27.019390+0000","last_fullsized":"2024-05-06T00:27:27.019390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:51:13.278704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":21017088,"num_objects":29,"num_object_clones":7,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":18,"num_read":1224,"num_read_kb":8333,"num_write":110,"num_write_kb":999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"51'224","reported_seq":5056,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-06T00:27:27.389594+0000","last_change":"2024-05-06T00:27:21.965729+0000","last_active":"2024-05-06T00:27:27.389594+0000","last_peered":"2024-05-06T00:27:27.389594+0000","last_clean":"2024-05-06T00:27:27.389594+0000","last_became_active":"2024-05-06T00:27:21.964410+0000","last_became_peered":"2024-05-06T00:27:21.964410+0000","last_unstale":"2024-05-06T00:27:27.389594+0000","last_undegraded":"2024-05-06T00:27:27.389594+0000","last_fullsized":"2024-05-06T00:27:27.389594+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:35:09.376083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":19230736,"num_objects":20,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":10,"num_read":5495,"num_read_kb":16564,"num_write":153,"num_write_kb":1274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":424892989,"num_objects":674,"num_object_clones":105,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":439,"num_read":56052,"num_read_kb":170666,"num_write":5614,"num_write_kb":63979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26255360,"data_stored":42480762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5408,"ondisk_log_size":5408,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640273,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37516,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334324,"statfs":{"total":96636764160,"available":96598347776,"internally_reserved":0,"allocated":2232320,"data_stored":2002218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640271,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41936,"kb_used_data":6600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329904,"statfs":{"total":96636764160,"available":96593821696,"internally_reserved":0,"allocated":6758400,"data_stored":10484365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640271,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42692,"kb_used_data":7352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329148,"statfs":{"total":96636764160,"available":96593047552,"internally_reserved":0,"allocated":7528448,"data_stored":11299807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640272,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44820,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327020,"statfs":{"total":96636764160,"available":96590868480,"internally_reserved":0,"allocated":5742592,"data_stored":5315604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640273,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":5688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":5824512,"data_stored":9467110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640271,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38080,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333760,"statfs":{"total":96636764160,"available":96597770240,"internally_reserved":0,"allocated":2805760,"data_stored":2362588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1650688,"data_stored":1605065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5246976,"data_stored":9239795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5165056,"data_stored":9121825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6356992,"data_stored":10482124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6180864,"data_stored":10257050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":1774903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:27:32.859 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-05-06T00:27:32.859 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:27:32.860 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-05-06T00:27:32.860 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-06T00:27:32.860 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-06T00:27:32.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:306 3.0 3:071dddc0:::rbd_data.110cd5ddb45c.000000000000000f:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.141704+0000", 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7250498759999999, 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.045510268, 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 306 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141704+0000", 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141704+0000", 2024-05-06T00:27:32.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141706+0000", 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.42e-06 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141713+0000", 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5599999999999996e-06 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141736+0000", 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2303e-05 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141753+0000", 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7329999999999998e-05 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142362+0000", 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060940099999999997 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142503+0000", 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000140142 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187215+0000", 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.044712111999999998 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:313 3.b 3:d733197f:::rbd_data.110cd5ddb45c.0000000000000013:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.144316+0000", 2024-05-06T00:27:32.871 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7224386210000002, 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019146326000000002, 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 313 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144316+0000", 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144316+0000", 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144317+0000", 2024-05-06T00:27:32.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.418e-06 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144323+0000", 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8089999999999998e-06 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144337+0000", 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.43e-05 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144346+0000", 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7940000000000008e-06 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.145272+0000", 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00092610399999999997 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.145397+0000", 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012485099999999999 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.163462+0000", 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018065049999999999 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:329 3.0 3:072d458a:::rbd_data.110cd5ddb45c.000000000000001b:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.144440+0000", 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7223144000000001, 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042898974999999999, 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 329 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144440+0000", 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144440+0000", 2024-05-06T00:27:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144440+0000", 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144443+0000", 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7480000000000002e-06 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144451+0000", 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4649999999999997e-06 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144456+0000", 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7869999999999998e-06 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144593+0000", 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000136997 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144913+0000", 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031916600000000001 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187339+0000", 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042426258000000001 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:347 3.4 3:23a228f1:::rbd_data.110cd5ddb45c.0000000000000024:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.146147+0000", 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.720606858, 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041542073999999998, 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 347 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.881 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146147+0000", 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146147+0000", 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146149+0000", 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3659999999999999e-06 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146156+0000", 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146176+0000", 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9327e-05 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146190+0000", 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4667e-05 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146368+0000", 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017755399999999999 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146885+0000", 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051747499999999999 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187690+0000", 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.040804171 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:364 3.0 3:05b36940:::rbd_data.110cd5ddb45c.000000000000002d:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.146227+0000", 2024-05-06T00:27:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7205277540000008, 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041576900999999999, 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 364 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146227+0000", 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146227+0000", 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146228+0000", 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146234+0000", 2024-05-06T00:27:32.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4030000000000004e-06 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146252+0000", 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7669999999999999e-05 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146267+0000", 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4853e-05 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.148565+0000", 2024-05-06T00:27:32.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022985319999999998 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.148874+0000", 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030827300000000002 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187803+0000", 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.038929962999999998 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:663 3.4 3:21bb1743:::rbd_data.110cf57807c6.000000000000000f:d [read 0~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159114+0000", 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7076401879999992, 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018070671, 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 663 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159114+0000", 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159114+0000", 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159115+0000", 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.266e-06 2024-05-06T00:27:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159125+0000", 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0057e-05 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159144+0000", 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8977000000000001e-05 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159175+0000", 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0883999999999998e-05 2024-05-06T00:27:32.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177079+0000", 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017904104000000001 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177123+0000", 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4054999999999997e-05 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177185+0000", 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1328000000000006e-05 2024-05-06T00:27:32.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:665 3.4 3:21bb1743:::rbd_data.110cf57807c6.000000000000000f:d [read 163840~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159215+0000", 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7075397270000003, 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018131736999999998, 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 665 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159215+0000", 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159215+0000", 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159216+0000", 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.246e-06 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159222+0000", 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1040000000000003e-06 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159240+0000", 2024-05-06T00:27:32.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8479999999999999e-05 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159255+0000", 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4674e-05 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177200+0000", 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017944985 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177256+0000", 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5498999999999998e-05 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177346+0000", 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0748999999999995e-05 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:674 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 573440~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159449+0000", 2024-05-06T00:27:32.898 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7073057249999994, 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018086707, 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 674 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159449+0000", 2024-05-06T00:27:32.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159449+0000", 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159450+0000", 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.043e-06 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159455+0000", 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1139999999999997e-06 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159472+0000", 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7513999999999999e-05 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159486+0000", 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3946e-05 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177372+0000", 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017885755999999999 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177447+0000", 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4933999999999998e-05 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177535+0000", 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8399999999999994e-05 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.902 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:675 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 737280~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159504+0000", 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7072507409999993, 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018168706999999999, 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 675 2024-05-06T00:27:32.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159504+0000", 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159504+0000", 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159505+0000", 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1400000000000001e-06 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159510+0000", 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.967e-06 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159528+0000", 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7943000000000001e-05 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159546+0000", 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.808e-05 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177551+0000", 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018004939000000001 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177608+0000", 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7839e-05 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177672+0000", 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3799000000000004e-05 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:676 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 901120~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159563+0000", 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7071909539999996, 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018312538999999999, 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 676 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159563+0000", 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159563+0000", 2024-05-06T00:27:32.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159565+0000", 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.15e-06 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159582+0000", 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6991e-05 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159601+0000", 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9449999999999998e-05 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159615+0000", 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4525999999999999e-05 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177696+0000", 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018080431000000001 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177771+0000", 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5226000000000001e-05 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177876+0000", 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104765 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:699 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 786432~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.160128+0000", 2024-05-06T00:27:32.911 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7066260819999997, 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018091955, 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 699 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160128+0000", 2024-05-06T00:27:32.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160128+0000", 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160130+0000", 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3239999999999999e-06 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160135+0000", 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3460000000000001e-06 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160154+0000", 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9040000000000001e-05 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160169+0000", 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.485e-05 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178058+0000", 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017889054000000001 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178132+0000", 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4173999999999998e-05 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178220+0000", 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8166999999999998e-05 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:701 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 950272~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.160187+0000", 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7065677780000001, 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018157427, 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 701 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160187+0000", 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160187+0000", 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160192+0000", 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6529999999999999e-06 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160198+0000", 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160317+0000", 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000119221 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160334+0000", 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7363999999999998e-05 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178235+0000", 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017901072000000001 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178277+0000", 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1628999999999997e-05 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178344+0000", 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6990999999999996e-05 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1036 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 720896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184560+0000", 2024-05-06T00:27:32.920 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.6821940849999999, 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018084947000000001, 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1036 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184560+0000", 2024-05-06T00:27:32.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184560+0000", 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184561+0000", 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.72e-07 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184562+0000", 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3570000000000001e-06 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184566+0000", 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.941e-06 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184569+0000", 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.145e-06 2024-05-06T00:27:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202533+0000", 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017963878999999999 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202581+0000", 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7920999999999997e-05 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202645+0000", 2024-05-06T00:27:32.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4431999999999996e-05 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1037 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184573+0000", 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.6821815529999999, 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018326584999999999, 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:32.925 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1037 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184573+0000", 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184573+0000", 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184573+0000", 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7000000000000001e-07 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184574+0000", 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.328e-06 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184578+0000", 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.946e-06 2024-05-06T00:27:32.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184581+0000", 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0910000000000001e-06 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202687+0000", 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018105857 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202768+0000", 2024-05-06T00:27:32.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1106000000000006e-05 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202899+0000", 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130987 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1047 3.0 3:07bcb856:::rbd_data.110c6c438b03.0000000000000001:11 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184585+0000", 2024-05-06T00:27:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.682169075, 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018905859000000001, 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1047 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184585+0000", 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184585+0000", 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184586+0000", 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7500000000000001e-07 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184587+0000", 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.373e-06 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184591+0000", 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0489999999999997e-06 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184594+0000", 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0699999999999998e-06 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203120+0000", 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018525498000000001 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203170+0000", 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0803999999999997e-05 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203491+0000", 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032079 2024-05-06T00:27:32.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1048 3.0 3:07bcb856:::rbd_data.110c6c438b03.0000000000000001:11 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184598+0000", 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.682156523, 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019327723000000002, 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1048 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184598+0000", 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184598+0000", 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184598+0000", 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7700000000000001e-07 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184600+0000", 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184604+0000", 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9899999999999999e-06 2024-05-06T00:27:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184607+0000", 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.084e-06 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203526+0000", 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018919642 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203604+0000", 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7653999999999999e-05 2024-05-06T00:27:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203926+0000", 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032150199999999999 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1049 3.0 3:07bcb856:::rbd_data.110c6c438b03.0000000000000001:11 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184610+0000", 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.6821438359999998, 2024-05-06T00:27:32.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019806744000000001, 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1049 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184610+0000", 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184610+0000", 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184611+0000", 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6800000000000002e-07 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184616+0000", 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2329999999999997e-06 2024-05-06T00:27:32.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184622+0000", 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9780000000000002e-06 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184625+0000", 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3450000000000002e-06 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203947+0000", 2024-05-06T00:27:32.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019322071999999999 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203995+0000", 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8055e-05 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204417+0000", 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042179300000000001 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1070 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184668+0000", 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.6820865980000002, 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019949142, 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1070 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.943 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184668+0000", 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184668+0000", 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184668+0000", 2024-05-06T00:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6100000000000002e-07 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184669+0000", 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.139e-06 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184675+0000", 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184679+0000", 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.912e-06 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204456+0000", 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019776789999999999 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204526+0000", 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9974000000000005e-05 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204617+0000", 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1181999999999995e-05 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1071 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184683+0000", 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.6820716170000001, 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020121881000000001, 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1071 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184683+0000", 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184683+0000", 2024-05-06T00:27:32.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184683+0000", 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7000000000000001e-07 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184684+0000", 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359999999999999e-06 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184688+0000", 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.941e-06 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184691+0000", 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2399999999999999e-06 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204648+0000", 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019957032999999999 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204718+0000", 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9474000000000006e-05 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204805+0000", 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6687000000000005e-05 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1072 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:32.951 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184695+0000", 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.6820592169999999, 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020306478999999999, 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1072 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184695+0000", 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184695+0000", 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184695+0000", 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6600000000000003e-07 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184697+0000", 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2839999999999999e-06 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184701+0000", 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9770000000000002e-06 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184704+0000", 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1829999999999998e-06 2024-05-06T00:27:32.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204836+0000", 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020131685 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204907+0000", 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1441999999999998e-05 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.205002+0000", 2024-05-06T00:27:32.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4642000000000006e-05 2024-05-06T00:27:32.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.956 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:32.956 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:32.956 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:32.958 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:32.958 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:32.958+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:32.959 INFO:teuthology.orchestra.run.smithi186.stdout:2032 write 0xb6be00 thru 0xb7b87a (0xfa7b bytes) 2024-05-06T00:27:32.959 INFO:teuthology.orchestra.run.smithi186.stdout:2033 write 0xc2e4000 thru 0xc2ec6f9 (0x86fa bytes) 2024-05-06T00:27:32.959 INFO:teuthology.orchestra.run.smithi186.stdout:2034 read 0xbe58000 thru 0xbe59136 (0x1137 bytes) 2024-05-06T00:27:32.959 INFO:teuthology.orchestra.run.smithi186.stdout:2035 read 0x9561800 thru 0x956da41 (0xc242 bytes) 2024-05-06T00:27:32.959 INFO:teuthology.orchestra.run.smithi186.stdout:2037 write 0xa299600 thru 0xa29b8f3 (0x22f4 bytes) 2024-05-06T00:27:32.959 INFO:teuthology.orchestra.run.smithi186.stdout:2040 write 0x3801200 thru 0x3806ff0 (0x5df1 bytes) 2024-05-06T00:27:32.959 INFO:teuthology.orchestra.run.smithi186.stdout:2041 write 0xaf69600 thru 0xaf70915 (0x7316 bytes) 2024-05-06T00:27:32.960 INFO:teuthology.orchestra.run.smithi186.stdout:2042 punch from 0xb434400 to 0xb439400, (0x5000 bytes) 2024-05-06T00:27:32.960 INFO:teuthology.orchestra.run.smithi186.stdout:2043 punch from 0x1f8a400 to 0x1f96a00, (0xc600 bytes) 2024-05-06T00:27:32.960 INFO:teuthology.orchestra.run.smithi186.stdout:2048 read 0x5a9d000 thru 0x5aaa9d6 (0xd9d7 bytes) 2024-05-06T00:27:32.960 INFO:teuthology.orchestra.run.smithi186.stdout:2050 write 0xce41e00 thru 0xce51340 (0xf541 bytes) 2024-05-06T00:27:32.962 INFO:teuthology.orchestra.run.smithi186.stdout:2051 read 0x2aa0600 thru 0x2aac7de (0xc1df bytes) 2024-05-06T00:27:32.962 INFO:teuthology.orchestra.run.smithi186.stdout:2053 write 0x3f0d000 thru 0x3f1ab3f (0xdb40 bytes) 2024-05-06T00:27:32.963 INFO:teuthology.orchestra.run.smithi186.stdout:2054 write 0x2b7e400 thru 0x2b87708 (0x9309 bytes) 2024-05-06T00:27:32.964 INFO:teuthology.orchestra.run.smithi186.stdout:2057 read 0x9d30a00 thru 0x9d34f41 (0x4542 bytes) 2024-05-06T00:27:32.964 INFO:teuthology.orchestra.run.smithi186.stdout:2058 punch from 0xcc16800 to 0xcc1b600, (0x4e00 bytes) 2024-05-06T00:27:32.968 INFO:teuthology.orchestra.run.smithi186.stdout:2059 punch from 0xf19200 to 0xf24800, (0xb600 bytes) 2024-05-06T00:27:32.970 INFO:teuthology.orchestra.run.smithi186.stdout:2060 read 0x35ee000 thru 0x35eec73 (0xc74 bytes) 2024-05-06T00:27:32.974 INFO:teuthology.orchestra.run.smithi186.stdout:2062 read 0x688ce00 thru 0x68913f0 (0x45f1 bytes) 2024-05-06T00:27:32.974 INFO:teuthology.orchestra.run.smithi186.stdout:2063 read 0x5151800 thru 0x515f671 (0xde72 bytes) 2024-05-06T00:27:32.975 INFO:teuthology.orchestra.run.smithi186.stdout:2064 read 0xb5a5a00 thru 0xb5ad9b0 (0x7fb1 bytes) 2024-05-06T00:27:32.975 INFO:teuthology.orchestra.run.smithi186.stdout:2065 write 0x6123c00 thru 0x6131a08 (0xde09 bytes) 2024-05-06T00:27:33.013 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-06T00:27:33.013 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:33.055 INFO:teuthology.orchestra.run.smithi186.stdout:2067 trunc from 0xce51400 to 0x4884e00 2024-05-06T00:27:33.058 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:33.058+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:33.135 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:33.136 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:33.136 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:33.136 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:33.142 INFO:teuthology.orchestra.run.smithi186.stdout:2068 write 0x84b2800 thru 0x84c1e9a (0xf69b bytes) 2024-05-06T00:27:33.144 INFO:teuthology.orchestra.run.smithi186.stdout:2069 write 0x4e43a00 thru 0x4e4ac05 (0x7206 bytes) 2024-05-06T00:27:33.145 INFO:teuthology.orchestra.run.smithi186.stdout:2071 punch from 0x609c200 to 0x60aa800, (0xe600 bytes) 2024-05-06T00:27:33.146 INFO:teuthology.orchestra.run.smithi186.stdout:2075 write 0x6edda00 thru 0x6ee4290 (0x6891 bytes) 2024-05-06T00:27:33.146 INFO:teuthology.orchestra.run.smithi186.stdout:2076 read 0xe5aa00 thru 0xe61633 (0x6c34 bytes) 2024-05-06T00:27:33.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:33.159 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:33.155+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:33.177 INFO:teuthology.orchestra.run.smithi186.stdout:2078 trunc from 0x84c2000 to 0x9a1600 2024-05-06T00:27:33.260 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:33.255+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:33.274 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:33.286 INFO:teuthology.orchestra.run.smithi186.stdout:2080 write 0xafda000 thru 0xafe13f6 (0x73f7 bytes) 2024-05-06T00:27:33.293 INFO:teuthology.orchestra.run.smithi186.stdout:2081 trunc from 0xafe1400 to 0x90a4600 2024-05-06T00:27:33.316 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:33.317 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:33.328 INFO:teuthology.orchestra.run.smithi186.stdout:2083 read 0x81aee00 thru 0x81b02bc (0x14bd bytes) 2024-05-06T00:27:33.332 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:33.332 INFO:teuthology.orchestra.run.smithi186.stdout:2085 flatten 2024-05-06T00:27:33.369 INFO:teuthology.orchestra.run.smithi186.stdout:2086 trunc from 0x90a4600 to 0x20bfe00 2024-05-06T00:27:33.377 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:33.378+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:33.388 INFO:teuthology.orchestra.run.smithi186.stdout:2087 punch from 0x1511400 to 0x151b200, (0x9e00 bytes) 2024-05-06T00:27:33.390 INFO:teuthology.orchestra.run.smithi186.stdout:2088 read 0x13c6800 thru 0x13d5372 (0xeb73 bytes) 2024-05-06T00:27:33.400 INFO:teuthology.orchestra.run.smithi186.stdout:2090 trunc from 0x20bfe00 to 0x5a58e00 2024-05-06T00:27:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:33.427 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:33.435 INFO:teuthology.orchestra.run.smithi186.stdout:2091 trunc from 0x5a58e00 to 0x97ba00 2024-05-06T00:27:33.445 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:33.445 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:33.461 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:33.462+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:33.464 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:33.464 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:33.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:33.487 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:33.489 INFO:teuthology.orchestra.run.smithi186.stdout:2092 write 0x6a8ac00 thru 0x6a8bd39 (0x113a bytes) 2024-05-06T00:27:33.495 INFO:teuthology.orchestra.run.smithi186.stdout:2095 trunc from 0x6a8be00 to 0x8bf9c00 2024-05-06T00:27:33.504 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:33.504 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:33.510 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:33.510 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:33.516 INFO:teuthology.orchestra.run.smithi186.stdout:2096 read 0x5d42200 thru 0x5d4aa07 (0x8808 bytes) 2024-05-06T00:27:33.518 INFO:teuthology.orchestra.run.smithi186.stdout:2097 read 0xf92c00 thru 0xf94aea (0x1eeb bytes) 2024-05-06T00:27:33.519 INFO:teuthology.orchestra.run.smithi186.stdout:2098 read 0x12cd200 thru 0x12d270f (0x5510 bytes) 2024-05-06T00:27:33.519 INFO:teuthology.orchestra.run.smithi186.stdout:2102 write 0x1347c00 thru 0x134eb14 (0x6f15 bytes) 2024-05-06T00:27:33.519 INFO:teuthology.orchestra.run.smithi186.stdout:2104 read 0x546cc00 thru 0x547bc50 (0xf051 bytes) 2024-05-06T00:27:33.519 INFO:teuthology.orchestra.run.smithi186.stdout:2107 write 0x42ac00 thru 0x438359 (0xd75a bytes) 2024-05-06T00:27:33.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:33.524 INFO:teuthology.orchestra.run.smithi186.stdout:2108 trunc from 0x8bf9c00 to 0xa257c00 2024-05-06T00:27:33.532 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:33.532 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:33.548 INFO:teuthology.orchestra.run.smithi186.stdout:2110 read 0x3ca1000 thru 0x3ca3573 (0x2574 bytes) 2024-05-06T00:27:33.550 INFO:teuthology.orchestra.run.smithi186.stdout:2111 write 0x4615600 thru 0x4615fa4 (0x9a5 bytes) 2024-05-06T00:27:33.550 INFO:teuthology.orchestra.run.smithi186.stdout:2113 write 0x7274000 thru 0x72827dd (0xe7de bytes) 2024-05-06T00:27:33.550 INFO:teuthology.orchestra.run.smithi186.stdout:2114 read 0x1394200 thru 0x13953ff (0x1200 bytes) 2024-05-06T00:27:33.550 INFO:teuthology.orchestra.run.smithi186.stdout:2116 read 0x5a36000 thru 0x5a3a66e (0x466f bytes) 2024-05-06T00:27:33.562 INFO:teuthology.orchestra.run.smithi186.stdout:2117 trunc from 0xa257c00 to 0xdd28800 2024-05-06T00:27:33.565 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:33.563+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:33.584 INFO:teuthology.orchestra.run.smithi186.stdout:2118 write 0x332e00 thru 0x334ae7 (0x1ce8 bytes) 2024-05-06T00:27:33.588 INFO:teuthology.orchestra.run.smithi186.stdout:2119 write 0xbfbba00 thru 0xbfc37ec (0x7ded bytes) 2024-05-06T00:27:33.588 INFO:teuthology.orchestra.run.smithi186.stdout:2123 read 0x5290200 thru 0x5298840 (0x8641 bytes) 2024-05-06T00:27:33.588 INFO:teuthology.orchestra.run.smithi186.stdout:2125 punch from 0x8660200 to 0x8662a00, (0x2800 bytes) 2024-05-06T00:27:33.589 INFO:teuthology.orchestra.run.smithi186.stdout:2126 punch from 0x184c00 to 0x188400, (0x3800 bytes) 2024-05-06T00:27:33.589 INFO:teuthology.orchestra.run.smithi186.stdout:2128 read 0x5dce800 thru 0x5dd6acb (0x82cc bytes) 2024-05-06T00:27:33.589 INFO:teuthology.orchestra.run.smithi186.stdout:2129 read 0x2bfa200 thru 0x2bfd55f (0x3360 bytes) 2024-05-06T00:27:33.600 INFO:teuthology.orchestra.run.smithi186.stdout:2130 trunc from 0xdd28800 to 0x7d78800 2024-05-06T00:27:33.628 INFO:teuthology.orchestra.run.smithi186.stdout:2131 write 0xbb600 thru 0xc6578 (0xaf79 bytes) 2024-05-06T00:27:33.633 INFO:teuthology.orchestra.run.smithi186.stdout:2132 read 0x729ce00 thru 0x729dc0e (0xe0f bytes) 2024-05-06T00:27:33.633 INFO:teuthology.orchestra.run.smithi186.stdout:2133 read 0x5742400 thru 0x57455d9 (0x31da bytes) 2024-05-06T00:27:33.633 INFO:teuthology.orchestra.run.smithi186.stdout:2134 read 0x371d000 thru 0x371e416 (0x1417 bytes) 2024-05-06T00:27:33.633 INFO:teuthology.orchestra.run.smithi186.stdout:2135 read 0x2a22400 thru 0x2a2765e (0x525f bytes) 2024-05-06T00:27:33.633 INFO:teuthology.orchestra.run.smithi186.stdout:2137 read 0x4ea3600 thru 0x4eaf62f (0xc030 bytes) 2024-05-06T00:27:33.664 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:33.659+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:33.665 INFO:teuthology.orchestra.run.smithi186.stdout:2139 write 0xaaea800 thru 0xaaf7009 (0xc80a bytes) 2024-05-06T00:27:33.667 INFO:teuthology.orchestra.run.smithi186.stdout:2140 read 0xa63cc00 thru 0xa63e7b7 (0x1bb8 bytes) 2024-05-06T00:27:33.667 INFO:teuthology.orchestra.run.smithi186.stdout:2141 read 0x9db4e00 thru 0x9db6154 (0x1355 bytes) 2024-05-06T00:27:33.667 INFO:teuthology.orchestra.run.smithi186.stdout:2142 read 0x2fa8600 thru 0x2fb1097 (0x8a98 bytes) 2024-05-06T00:27:33.667 INFO:teuthology.orchestra.run.smithi186.stdout:2143 read 0x82cea00 thru 0x82d5943 (0x6f44 bytes) 2024-05-06T00:27:33.668 INFO:teuthology.orchestra.run.smithi186.stdout:2144 trunc from 0xaaf7200 to 0xa756200 2024-05-06T00:27:33.668 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:33.668 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:33.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:33.730 INFO:teuthology.orchestra.run.smithi186.stdout:2145 write 0xaf9ca00 thru 0xafa5227 (0x8828 bytes) 2024-05-06T00:27:33.731 INFO:teuthology.orchestra.run.smithi186.stdout:2146 write 0x4629a00 thru 0x46376a0 (0xdca1 bytes) 2024-05-06T00:27:33.765 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:33.766+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:33.775 INFO:teuthology.orchestra.run.smithi186.stdout:2148 write 0xcfb3200 thru 0xcfb6c52 (0x3a53 bytes) 2024-05-06T00:27:33.777 INFO:teuthology.orchestra.run.smithi186.stdout:2152 punch from 0xc273e00 to 0xc275c00, (0x1e00 bytes) 2024-05-06T00:27:33.796 INFO:teuthology.orchestra.run.smithi186.stdout:2155 trunc from 0xcfb6e00 to 0x38f0c00 2024-05-06T00:27:33.853 INFO:teuthology.orchestra.run.smithi186.stdout:2156 trunc from 0x38f0c00 to 0x71bdc00 2024-05-06T00:27:33.860 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:33.860 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:33.866 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:33.863+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:33.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:33.880 INFO:teuthology.orchestra.run.smithi186.stdout:2158 read 0xc75400 thru 0xc838c0 (0xe4c1 bytes) 2024-05-06T00:27:33.883 INFO:teuthology.orchestra.run.smithi186.stdout:2160 punch from 0x64ea200 to 0x64f1a00, (0x7800 bytes) 2024-05-06T00:27:33.913 INFO:teuthology.orchestra.run.smithi186.stdout:2162 write 0x9034600 thru 0x904044c (0xbe4d bytes) 2024-05-06T00:27:33.914 INFO:teuthology.orchestra.run.smithi186.stdout:2165 punch from 0x5e96400 to 0x5ea5800, (0xf400 bytes) 2024-05-06T00:27:33.915 INFO:teuthology.orchestra.run.smithi186.stdout:2166 read 0x2f80400 thru 0x2f8fad6 (0xf6d7 bytes) 2024-05-06T00:27:33.915 INFO:teuthology.orchestra.run.smithi186.stdout:2167 punch from 0x1a4e000 to 0x1a5b000, (0xd000 bytes) 2024-05-06T00:27:33.916 INFO:teuthology.orchestra.run.smithi186.stdout:2169 read 0x8c2d600 thru 0x8c35f4c (0x894d bytes) 2024-05-06T00:27:33.916 INFO:teuthology.orchestra.run.smithi186.stdout:2170 punch from 0xdd8a00 to 0xdd8e00, (0x400 bytes) 2024-05-06T00:27:33.916 INFO:teuthology.orchestra.run.smithi186.stdout:2171 read 0x352e800 thru 0x353dd1e (0xf51f bytes) 2024-05-06T00:27:33.916 INFO:teuthology.orchestra.run.smithi186.stdout:2172 punch from 0x618d600 to 0x6198c00, (0xb600 bytes) 2024-05-06T00:27:33.917 INFO:teuthology.orchestra.run.smithi186.stdout:2177 write 0x4a22200 thru 0x4a2d322 (0xb123 bytes) 2024-05-06T00:27:33.917 INFO:teuthology.orchestra.run.smithi186.stdout:2179 read 0x13ca800 thru 0x13cc684 (0x1e85 bytes) 2024-05-06T00:27:33.917 INFO:teuthology.orchestra.run.smithi186.stdout:2180 read 0x820ca00 thru 0x8210f8b (0x458c bytes) 2024-05-06T00:27:33.917 INFO:teuthology.orchestra.run.smithi186.stdout:2181 write 0x4986c00 thru 0x498db98 (0x6f99 bytes) 2024-05-06T00:27:33.926 INFO:teuthology.orchestra.run.smithi186.stdout:2183 trunc from 0x9040600 to 0xdc75c00 2024-05-06T00:27:33.960 INFO:teuthology.orchestra.run.smithi186.stdout:2184 read 0xa759400 thru 0xa75981d (0x41e bytes) 2024-05-06T00:27:33.964 INFO:teuthology.orchestra.run.smithi186.stdout:2186 read 0xaaafe00 thru 0xaab8c6b (0x8e6c bytes) 2024-05-06T00:27:33.964 INFO:teuthology.orchestra.run.smithi186.stdout:2187 write 0x3852e00 thru 0x385f05d (0xc25e bytes) 2024-05-06T00:27:33.966 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:33.963+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:34.004 INFO:teuthology.orchestra.run.smithi186.stdout:2189 trunc from 0xdc75c00 to 0x1c800 2024-05-06T00:27:34.052 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:34.052 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:34.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:34.068 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:34.066+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:34.069 INFO:teuthology.orchestra.run.smithi186.stdout:2190 read 0x9c00 thru 0x16964 (0xcd65 bytes) 2024-05-06T00:27:34.074 INFO:teuthology.orchestra.run.smithi186.stdout:2191 punch from 0xca00 to 0xde00, (0x1400 bytes) 2024-05-06T00:27:34.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:27:34.132 INFO:teuthology.orchestra.run.smithi186.stdout:2192 write 0xd8b7400 thru 0xd8bc911 (0x5512 bytes) 2024-05-06T00:27:34.132 INFO:teuthology.orchestra.run.smithi186.stdout:2194 write 0x1ab9400 thru 0x1ac1e50 (0x8a51 bytes) 2024-05-06T00:27:34.132 INFO:teuthology.orchestra.run.smithi186.stdout:2196 punch from 0x8f7e200 to 0x8f80400, (0x2200 bytes) 2024-05-06T00:27:34.133 INFO:teuthology.orchestra.run.smithi186.stdout:2198 write 0xca9ce00 thru 0xcaa0fc0 (0x41c1 bytes) 2024-05-06T00:27:34.134 INFO:teuthology.orchestra.run.smithi186.stdout:2200 write 0x6926e00 thru 0x692820a (0x140b bytes) 2024-05-06T00:27:34.134 INFO:teuthology.orchestra.run.smithi186.stdout:2202 write 0x971f000 thru 0x972bdef (0xcdf0 bytes) 2024-05-06T00:27:34.134 INFO:teuthology.orchestra.run.smithi186.stdout:2204 write 0x11fa800 thru 0x11fe624 (0x3e25 bytes) 2024-05-06T00:27:34.169 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:34.170+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:34.175 INFO:teuthology.orchestra.run.smithi186.stdout:2205 write 0xe092000 thru 0xe0982d3 (0x62d4 bytes) 2024-05-06T00:27:34.176 INFO:teuthology.orchestra.run.smithi186.stdout:2206 read 0x42c6000 thru 0x42d5ddf (0xfde0 bytes) 2024-05-06T00:27:34.177 INFO:teuthology.orchestra.run.smithi186.stdout:2207 write 0xa83fc00 thru 0xa849c2f (0xa030 bytes) 2024-05-06T00:27:34.177 INFO:teuthology.orchestra.run.smithi186.stdout:2208 punch from 0x2fdd600 to 0x2fe4400, (0x6e00 bytes) 2024-05-06T00:27:34.181 INFO:teuthology.orchestra.run.smithi186.stdout:2209 trunc from 0xe098400 to 0xc860600 2024-05-06T00:27:34.229 INFO:teuthology.orchestra.run.smithi186.stdout:2210 write 0xe371400 thru 0xe372509 (0x110a bytes) 2024-05-06T00:27:34.230 INFO:teuthology.orchestra.run.smithi186.stdout:2211 punch from 0xe15fc00 to 0xe168800, (0x8c00 bytes) 2024-05-06T00:27:34.231 INFO:teuthology.orchestra.run.smithi186.stdout:2212 read 0x64a7a00 thru 0x64abbe1 (0x41e2 bytes) 2024-05-06T00:27:34.231 INFO:teuthology.orchestra.run.smithi186.stdout:2214 read 0x967fe00 thru 0x968aca3 (0xaea4 bytes) 2024-05-06T00:27:34.250 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:34.251 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:34.253 INFO:teuthology.orchestra.run.smithi186.stdout:2215 trunc from 0xe372600 to 0x62d4400 2024-05-06T00:27:34.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:34.269 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:34.270+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:34.279 INFO:teuthology.orchestra.run.smithi186.stdout:2217 write 0x4ed3a00 thru 0x4ee1561 (0xdb62 bytes) 2024-05-06T00:27:34.281 INFO:teuthology.orchestra.run.smithi186.stdout:2221 read 0x2bc2600 thru 0x2bc273a (0x13b bytes) 2024-05-06T00:27:34.282 INFO:teuthology.orchestra.run.smithi186.stdout:2222 read 0x2a76200 thru 0x2a83802 (0xd603 bytes) 2024-05-06T00:27:34.291 INFO:teuthology.orchestra.run.smithi186.stdout:2224 trunc from 0x62d4400 to 0xadeb400 2024-05-06T00:27:34.329 INFO:teuthology.orchestra.run.smithi186.stdout:2226 write 0xe3dc200 thru 0xe3dc461 (0x262 bytes) 2024-05-06T00:27:34.343 INFO:teuthology.orchestra.run.smithi186.stdout:2229 trunc from 0xe3dc600 to 0x615bc00 2024-05-06T00:27:34.371 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:34.370+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:34.379 INFO:teuthology.orchestra.run.smithi186.stdout:2233 read 0x3b6a600 thru 0x3b785d0 (0xdfd1 bytes) 2024-05-06T00:27:34.381 INFO:teuthology.orchestra.run.smithi186.stdout:2234 read 0x2ac8400 thru 0x2ac8dff (0xa00 bytes) 2024-05-06T00:27:34.382 INFO:teuthology.orchestra.run.smithi186.stdout:2236 read 0x24c9a00 thru 0x24d7617 (0xdc18 bytes) 2024-05-06T00:27:34.384 INFO:teuthology.orchestra.run.smithi186.stdout:2237 punch from 0x5c59a00 to 0x5c5b800, (0x1e00 bytes) 2024-05-06T00:27:34.384 INFO:teuthology.orchestra.run.smithi186.stdout:2238 read 0x4190e00 thru 0x41a0a0f (0xfc10 bytes) 2024-05-06T00:27:34.398 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:34.399 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:34.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:34.422 INFO:teuthology.orchestra.run.smithi186.stdout:2239 write 0xe0b3a00 thru 0xe0b9343 (0x5944 bytes) 2024-05-06T00:27:34.424 INFO:teuthology.orchestra.run.smithi186.stdout:2240 read 0x400a00 thru 0x4052a0 (0x48a1 bytes) 2024-05-06T00:27:34.425 INFO:teuthology.orchestra.run.smithi186.stdout:2241 write 0xefec00 thru 0xf039d6 (0x4dd7 bytes) 2024-05-06T00:27:34.425 INFO:teuthology.orchestra.run.smithi186.stdout:2245 write 0xcfa4600 thru 0xcfa4fd8 (0x9d9 bytes) 2024-05-06T00:27:34.425 INFO:teuthology.orchestra.run.smithi186.stdout:2246 clone 16 order 18 su 1024 sc 10 2024-05-06T00:27:34.472 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:34.467+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:34.534 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:34.534 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:34.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:34.573 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:34.574+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:34.674 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:34.671+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:34.712 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:34.712 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:34.725 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:34.813 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:34.811+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:34.876 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:34.871+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:34.895 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:34.896 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:34.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:34.977 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:34.978+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:27:35.078 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:35.078 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:35.078 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:35.078+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:35.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:35.179 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:35.175+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:35.260 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:35.260 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:35.274 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:35.321 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:35.322+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:35.384 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:35.379+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:35.395 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:35.395 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:35.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:35.503 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:35.502+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:35.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:35.578 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:35.578 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:35.582 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:35.582+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:35.593 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:35.685 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:35.679+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:35.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:35.736 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:35.745 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-05-06T00:27:35.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:35.754 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:35.768 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:35.769 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:35.777 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:35.777 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:35.784 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:35.786 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:35.783+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:35.798 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:35.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:35.834 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:35.901 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:35.902+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:35.929 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:35.930 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:35.945 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:35.989 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:35.990+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:36.089 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:36.087+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:36.122 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:306 3.0 3:071dddc0:::rbd_data.110cd5ddb45c.000000000000000f:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.141704+0000", 2024-05-06T00:27:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.978713278000001, 2024-05-06T00:27:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.045510268, 2024-05-06T00:27:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 306 2024-05-06T00:27:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141704+0000", 2024-05-06T00:27:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141704+0000", 2024-05-06T00:27:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141706+0000", 2024-05-06T00:27:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.42e-06 2024-05-06T00:27:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141713+0000", 2024-05-06T00:27:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5599999999999996e-06 2024-05-06T00:27:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141736+0000", 2024-05-06T00:27:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2303e-05 2024-05-06T00:27:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.141753+0000", 2024-05-06T00:27:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7329999999999998e-05 2024-05-06T00:27:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142362+0000", 2024-05-06T00:27:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060940099999999997 2024-05-06T00:27:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.142503+0000", 2024-05-06T00:27:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000140142 2024-05-06T00:27:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187215+0000", 2024-05-06T00:27:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.044712111999999998 2024-05-06T00:27:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:313 3.b 3:d733197f:::rbd_data.110cd5ddb45c.0000000000000013:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.144316+0000", 2024-05-06T00:27:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.976102022999999, 2024-05-06T00:27:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019146326000000002, 2024-05-06T00:27:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 313 2024-05-06T00:27:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144316+0000", 2024-05-06T00:27:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144316+0000", 2024-05-06T00:27:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144317+0000", 2024-05-06T00:27:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.418e-06 2024-05-06T00:27:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144323+0000", 2024-05-06T00:27:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8089999999999998e-06 2024-05-06T00:27:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144337+0000", 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.43e-05 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144346+0000", 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7940000000000008e-06 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.145272+0000", 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00092610399999999997 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.145397+0000", 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012485099999999999 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.163462+0000", 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018065049999999999 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:329 3.0 3:072d458a:::rbd_data.110cd5ddb45c.000000000000001b:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.144440+0000", 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.975977801999999, 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042898974999999999, 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 329 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144440+0000", 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144440+0000", 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144440+0000", 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:27:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144443+0000", 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7480000000000002e-06 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144451+0000", 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4649999999999997e-06 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144456+0000", 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7869999999999998e-06 2024-05-06T00:27:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144593+0000", 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000136997 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.144913+0000", 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031916600000000001 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187339+0000", 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042426258000000001 2024-05-06T00:27:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:347 3.4 3:23a228f1:::rbd_data.110cd5ddb45c.0000000000000024:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.146147+0000", 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.974270260000001, 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041542073999999998, 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 347 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146147+0000", 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146147+0000", 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146149+0000", 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3659999999999999e-06 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146156+0000", 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:27:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146176+0000", 2024-05-06T00:27:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9327e-05 2024-05-06T00:27:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146190+0000", 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4667e-05 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146368+0000", 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017755399999999999 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146885+0000", 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051747499999999999 2024-05-06T00:27:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187690+0000", 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.040804171 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:364 3.0 3:05b36940:::rbd_data.110cd5ddb45c.000000000000002d:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.146227+0000", 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.974191156, 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041576900999999999, 2024-05-06T00:27:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 364 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146227+0000", 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146227+0000", 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146228+0000", 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146234+0000", 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4030000000000004e-06 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146252+0000", 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7669999999999999e-05 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.146267+0000", 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4853e-05 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.148565+0000", 2024-05-06T00:27:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022985319999999998 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.148874+0000", 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030827300000000002 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.187803+0000", 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.038929962999999998 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:663 3.4 3:21bb1743:::rbd_data.110cf57807c6.000000000000000f:d [read 0~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159114+0000", 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.96130359, 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018070671, 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 663 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159114+0000", 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159114+0000", 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159115+0000", 2024-05-06T00:27:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.266e-06 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159125+0000", 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0057e-05 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159144+0000", 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8977000000000001e-05 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159175+0000", 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0883999999999998e-05 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177079+0000", 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017904104000000001 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177123+0000", 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4054999999999997e-05 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177185+0000", 2024-05-06T00:27:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1328000000000006e-05 2024-05-06T00:27:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:665 3.4 3:21bb1743:::rbd_data.110cf57807c6.000000000000000f:d [read 163840~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159215+0000", 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.961203128999999, 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018131736999999998, 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 665 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159215+0000", 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159215+0000", 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159216+0000", 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.246e-06 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159222+0000", 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1040000000000003e-06 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159240+0000", 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8479999999999999e-05 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159255+0000", 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4674e-05 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177200+0000", 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017944985 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177256+0000", 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5498999999999998e-05 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177346+0000", 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0748999999999995e-05 2024-05-06T00:27:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:674 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 573440~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159449+0000", 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.960969127, 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018086707, 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 674 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159449+0000", 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159449+0000", 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159450+0000", 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.043e-06 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159455+0000", 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1139999999999997e-06 2024-05-06T00:27:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159472+0000", 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7513999999999999e-05 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159486+0000", 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3946e-05 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177372+0000", 2024-05-06T00:27:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017885755999999999 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177447+0000", 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4933999999999998e-05 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177535+0000", 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8399999999999994e-05 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:675 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 737280~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159504+0000", 2024-05-06T00:27:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.960914143, 2024-05-06T00:27:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018168706999999999, 2024-05-06T00:27:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 675 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159504+0000", 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159504+0000", 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159505+0000", 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1400000000000001e-06 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159510+0000", 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.967e-06 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159528+0000", 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7943000000000001e-05 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159546+0000", 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.808e-05 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177551+0000", 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018004939000000001 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177608+0000", 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7839e-05 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177672+0000", 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3799000000000004e-05 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:676 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 901120~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.159563+0000", 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.960854356, 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018312538999999999, 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 676 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159563+0000", 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159563+0000", 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159565+0000", 2024-05-06T00:27:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.15e-06 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159582+0000", 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6991e-05 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159601+0000", 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9449999999999998e-05 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.159615+0000", 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4525999999999999e-05 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177696+0000", 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018080431000000001 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177771+0000", 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5226000000000001e-05 2024-05-06T00:27:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.177876+0000", 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104765 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:699 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 786432~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.160128+0000", 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.960289484, 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018091955, 2024-05-06T00:27:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 699 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160128+0000", 2024-05-06T00:27:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160128+0000", 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160130+0000", 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3239999999999999e-06 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160135+0000", 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3460000000000001e-06 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160154+0000", 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9040000000000001e-05 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160169+0000", 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.485e-05 2024-05-06T00:27:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178058+0000", 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017889054000000001 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178132+0000", 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4173999999999998e-05 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178220+0000", 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8166999999999998e-05 2024-05-06T00:27:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4932.0:701 3.4 3:24b38e32:::rbd_data.110cf57807c6.0000000000000015:d [read 950272~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:24.160187+0000", 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.960231179999999, 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018157427, 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4932", 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 701 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160187+0000", 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160187+0000", 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160192+0000", 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6529999999999999e-06 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160198+0000", 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160317+0000", 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000119221 2024-05-06T00:27:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.160334+0000", 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7363999999999998e-05 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178235+0000", 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017901072000000001 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178277+0000", 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1628999999999997e-05 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:24.178344+0000", 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6990999999999996e-05 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1036 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 720896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184560+0000", 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9358574869999998, 2024-05-06T00:27:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018084947000000001, 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1036 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184560+0000", 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184560+0000", 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184561+0000", 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.72e-07 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184562+0000", 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3570000000000001e-06 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184566+0000", 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.941e-06 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184569+0000", 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.145e-06 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202533+0000", 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017963878999999999 2024-05-06T00:27:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202581+0000", 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7920999999999997e-05 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202645+0000", 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4431999999999996e-05 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1037 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184573+0000", 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9358449550000003, 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018326584999999999, 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1037 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184573+0000", 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184573+0000", 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184573+0000", 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7000000000000001e-07 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184574+0000", 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.328e-06 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184578+0000", 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.946e-06 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184581+0000", 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0910000000000001e-06 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202687+0000", 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018105857 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202768+0000", 2024-05-06T00:27:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1106000000000006e-05 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.202899+0000", 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130987 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1047 3.0 3:07bcb856:::rbd_data.110c6c438b03.0000000000000001:11 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184585+0000", 2024-05-06T00:27:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9358324769999999, 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018905859000000001, 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1047 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184585+0000", 2024-05-06T00:27:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184585+0000", 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184586+0000", 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7500000000000001e-07 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184587+0000", 2024-05-06T00:27:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.373e-06 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184591+0000", 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0489999999999997e-06 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184594+0000", 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0699999999999998e-06 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203120+0000", 2024-05-06T00:27:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018525498000000001 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203170+0000", 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0803999999999997e-05 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203491+0000", 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032079 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1048 3.0 3:07bcb856:::rbd_data.110c6c438b03.0000000000000001:11 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184598+0000", 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9358199249999997, 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019327723000000002, 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1048 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184598+0000", 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184598+0000", 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184598+0000", 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7700000000000001e-07 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184600+0000", 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184604+0000", 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9899999999999999e-06 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184607+0000", 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.084e-06 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203526+0000", 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018919642 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203604+0000", 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7653999999999999e-05 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203926+0000", 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032150199999999999 2024-05-06T00:27:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1049 3.0 3:07bcb856:::rbd_data.110c6c438b03.0000000000000001:11 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184610+0000", 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9358072379999998, 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019806744000000001, 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1049 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184610+0000", 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184610+0000", 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184611+0000", 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6800000000000002e-07 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184616+0000", 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2329999999999997e-06 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184622+0000", 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9780000000000002e-06 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184625+0000", 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3450000000000002e-06 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203947+0000", 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019322071999999999 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.203995+0000", 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8055e-05 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204417+0000", 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042179300000000001 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1070 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184668+0000", 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9357499999999996, 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019949142, 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1070 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184668+0000", 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184668+0000", 2024-05-06T00:27:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184668+0000", 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6100000000000002e-07 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184669+0000", 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.139e-06 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184675+0000", 2024-05-06T00:27:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184679+0000", 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.912e-06 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204456+0000", 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019776789999999999 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204526+0000", 2024-05-06T00:27:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9974000000000005e-05 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204617+0000", 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1181999999999995e-05 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1071 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184683+0000", 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.935735019, 2024-05-06T00:27:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020121881000000001, 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1071 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184683+0000", 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184683+0000", 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184683+0000", 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7000000000000001e-07 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184684+0000", 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359999999999999e-06 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184688+0000", 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.941e-06 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184691+0000", 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2399999999999999e-06 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204648+0000", 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019957032999999999 2024-05-06T00:27:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204718+0000", 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9474000000000006e-05 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204805+0000", 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6687000000000005e-05 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5043.0:1072 3.0 3:01d38f40:::rbd_data.110c6c438b03.0000000000000008:11 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:31.184695+0000", 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9357226189999999, 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020306478999999999, 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5043", 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3715577530", 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1072 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184695+0000", 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184695+0000", 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184695+0000", 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6600000000000003e-07 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184697+0000", 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2839999999999999e-06 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184701+0000", 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9770000000000002e-06 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.184704+0000", 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1829999999999998e-06 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204836+0000", 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020131685 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.204907+0000", 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1441999999999998e-05 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:31.205002+0000", 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4642000000000006e-05 2024-05-06T00:27:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.223 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:36.224 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:36.225 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:36.219+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:36.325 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:36.326+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:36.363 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:36.363 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:36.377 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:36.426 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:36.423+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:36.527 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:36.523+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:36.565 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:36.566 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:36.566 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:36.566 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:36.566 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:36.582 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:36.628 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:36.623+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:36.715 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:36.715 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:36.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:36.715 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:36.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.715 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:858 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2752512~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005516+0000", 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7096809270000008, 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054192616999999998, 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 858 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.716 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005516+0000", 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005516+0000", 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005517+0000", 2024-05-06T00:27:36.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0669999999999999e-06 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005522+0000", 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005539+0000", 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6929999999999999e-05 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005554+0000", 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4856e-05 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.059256+0000", 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053702568999999999 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.059354+0000", 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8048999999999996e-05 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.059708+0000", 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035396599999999999 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.720 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:859 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3211264~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005624+0000", 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7095729360000007, 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054575108999999997, 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 859 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005624+0000", 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005624+0000", 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005625+0000", 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7129999999999999e-06 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005632+0000", 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6800000000000004e-06 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005652+0000", 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0327e-05 2024-05-06T00:27:36.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005667+0000", 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.505e-05 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.059760+0000", 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054092619000000002 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.059865+0000", 2024-05-06T00:27:36.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105134 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060199+0000", 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033358599999999997 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:860 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3407872~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7095048859999995, 2024-05-06T00:27:36.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054999124000000003, 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 860 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:36.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005693+0000", 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3969999999999999e-06 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005699+0000", 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1399999999999997e-06 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005718+0000", 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9190000000000001e-05 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005733+0000", 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4303e-05 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060250+0000", 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054516805000000002 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060353+0000", 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103509 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060691+0000", 2024-05-06T00:27:36.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033777999999999998 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:861 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7094442050000005, 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055437840000000002, 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 861 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005754+0000", 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005773+0000", 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9817e-05 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005792+0000", 2024-05-06T00:27:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8770000000000002e-05 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005824+0000", 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1643000000000002e-05 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060742+0000", 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054918417999999997 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060845+0000", 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102507 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061190+0000", 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034541500000000001 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.735 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:862 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4063232~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.735 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:36.735 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7093507280000004, 2024-05-06T00:27:36.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055843853999999998, 2024-05-06T00:27:36.735 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 862 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005847+0000", 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3200000000000001e-06 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005854+0000", 2024-05-06T00:27:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8299999999999998e-06 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005872+0000", 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.84e-05 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005886+0000", 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.347e-05 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061241+0000", 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055355559999999998 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061343+0000", 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101335 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061690+0000", 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034693899999999999 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.739 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:863 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.709290631, 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056056918999999997, 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 863 2024-05-06T00:27:36.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005907+0000", 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.234e-06 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005913+0000", 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8000000000000004e-06 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005932+0000", 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.895e-05 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005964+0000", 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2016e-05 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061741+0000", 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055776824000000003 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061840+0000", 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9469000000000006e-05 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061963+0000", 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000122626 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:899 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2392064~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.744 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7092125110000005, 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056211016000000003, 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 899 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005985+0000", 2024-05-06T00:27:36.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:36.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005992+0000", 2024-05-06T00:27:36.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-06T00:27:36.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006011+0000", 2024-05-06T00:27:36.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8669999999999999e-05 2024-05-06T00:27:36.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006024+0000", 2024-05-06T00:27:36.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-05 2024-05-06T00:27:36.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062007+0000", 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055982453000000001 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062096+0000", 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9549000000000006e-05 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062195+0000", 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8999999999999994e-05 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:900 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2850816~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7091530800000001, 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056426437000000003, 2024-05-06T00:27:36.751 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.752 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.752 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.752 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.752 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 900 2024-05-06T00:27:36.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.753 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:36.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:36.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006045+0000", 2024-05-06T00:27:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.22e-06 2024-05-06T00:27:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006064+0000", 2024-05-06T00:27:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8997000000000001e-05 2024-05-06T00:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006083+0000", 2024-05-06T00:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9449999999999998e-05 2024-05-06T00:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006098+0000", 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.451e-05 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062245+0000", 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056147619000000003 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062346+0000", 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100541 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062470+0000", 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012410000000000001 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:901 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7090789999999991, 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056582556999999999, 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 901 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006119+0000", 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006125+0000", 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.207e-06 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006143+0000", 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.821e-05 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006157+0000", 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4197e-05 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062513+0000", 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056355322999999999 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062600+0000", 2024-05-06T00:27:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7060999999999999e-05 2024-05-06T00:27:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062700+0000", 2024-05-06T00:27:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100369 2024-05-06T00:27:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:902 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7089495590000006, 2024-05-06T00:27:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056710658999999997, 2024-05-06T00:27:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.767 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.767 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 902 2024-05-06T00:27:36.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.767 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:36.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:36.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006249+0000", 2024-05-06T00:27:36.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5939999999999999e-06 2024-05-06T00:27:36.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006256+0000", 2024-05-06T00:27:36.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1029999999999996e-06 2024-05-06T00:27:36.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006277+0000", 2024-05-06T00:27:36.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1637e-05 2024-05-06T00:27:36.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006293+0000", 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.525e-05 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062749+0000", 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056456081999999998 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062845+0000", 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6194999999999996e-05 2024-05-06T00:27:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062958+0000", 2024-05-06T00:27:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000112798 2024-05-06T00:27:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:903 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3702784~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7088818339999996, 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056894469000000003, 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 903 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.775 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006316+0000", 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2759999999999999e-06 2024-05-06T00:27:36.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006322+0000", 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9839999999999998e-06 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006340+0000", 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8465999999999999e-05 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006354+0000", 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3137e-05 2024-05-06T00:27:36.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063007+0000", 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056652965 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063100+0000", 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3638e-05 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063209+0000", 2024-05-06T00:27:36.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109003 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:904 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4161536~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7088227909999993, 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057092006000000001, 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.779 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 904 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006375+0000", 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006381+0000", 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0700000000000003e-06 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006399+0000", 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8372999999999998e-05 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006413+0000", 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3917000000000001e-05 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063258+0000", 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056844680000000002 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063352+0000", 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3991000000000002e-05 2024-05-06T00:27:36.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063466+0000", 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113768 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:905 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4358144~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7087651269999995, 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057200123999999998, 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 905 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006433+0000", 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006438+0000", 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5770000000000001e-06 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006456+0000", 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006469+0000", 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2666999999999999e-05 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063509+0000", 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057039566999999999 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063563+0000", 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4360000000000001e-05 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063632+0000", 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8663000000000003e-05 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:921 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7087098730000001, 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057645638999999999, 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 921 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006488+0000", 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0699999999999999e-06 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006493+0000", 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2159999999999997e-06 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006511+0000", 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8454e-05 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006525+0000", 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.342e-05 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063673+0000", 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057148440000000002 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063765+0000", 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1608999999999997e-05 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064132+0000", 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036743000000000002 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:922 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2785280~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7086543299999999, 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058081037000000002, 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 922 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006543+0000", 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.127e-06 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006568+0000", 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4777e-05 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006588+0000", 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0203999999999998e-05 2024-05-06T00:27:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006616+0000", 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7625999999999999e-05 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064169+0000", 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057553460000000001 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064264+0000", 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4773999999999998e-05 2024-05-06T00:27:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064623+0000", 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000359069 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:923 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3244032~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7085591959999995, 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058440913999999997, 2024-05-06T00:27:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 923 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006639+0000", 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006657+0000", 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8692999999999999e-05 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006677+0000", 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9760999999999999e-05 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006692+0000", 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4467000000000001e-05 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064658+0000", 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057966692 2024-05-06T00:27:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064746+0000", 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7851999999999998e-05 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065078+0000", 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033224200000000001 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7084854949999997, 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7082569230000004, 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7081796390000008, 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.809 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:36.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:27:36.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:27:36.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7079836579999998, 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:27:36.813 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:27:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:36.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:36.819 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:36.815+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:36.921 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:36.918+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:36.954 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:36.954 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:36.954 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:36.954 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:36.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:37.021 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:37.019+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:37.091 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:37.091 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:37.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:27:37.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:37.125 INFO:teuthology.orchestra.run.smithi186.stdout:2247 trunc from 0xe0b9400 to 0x4c3a00 2024-05-06T00:27:37.137 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:37.138+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:37.223 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:37.219+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:37.249 INFO:teuthology.orchestra.run.smithi186.stdout:2248 punch from 0x264e00 to 0x266a00, (0x1c00 bytes) 2024-05-06T00:27:37.276 INFO:teuthology.orchestra.run.smithi186.stdout:2250 punch from 0x36c800 to 0x371a00, (0x5200 bytes) 2024-05-06T00:27:37.287 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:37.287 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:37.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:37.288 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:37.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.288 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:37.288 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:37.288 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.682642180000002, 2024-05-06T00:27:37.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:37.290 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:37.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.290 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:37.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:37.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:37.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:37.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:37.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:37.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:37.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:37.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:37.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:37.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:37.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:37.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:37.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:37.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:37.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:37.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:37.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.299 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409993481000001, 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:27:37.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:27:37.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:27:37.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:27:37.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409958305, 2024-05-06T00:27:37.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:27:37.307 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.307 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.307 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:27:37.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:27:37.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:27:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:27:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:27:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:27:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:27:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:27:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409869625999999, 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:27:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409767682999998, 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:27:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409658580999999, 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:27:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409560721999998, 2024-05-06T00:27:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:27:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409495671999998, 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:27:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:27:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409403026, 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:27:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:27:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.409331022, 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:27:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:27:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649869536000001, 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:27:37.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:27:37.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649850288, 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:27:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649800167999999, 2024-05-06T00:27:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:27:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:27:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:27:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649779117000001, 2024-05-06T00:27:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:27:37.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649763797999999, 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:27:37.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:27:37.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649746916000002, 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:37.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:27:37.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:27:37.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.395 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649728580000001, 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:27:37.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:27:37.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:27:37.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649712577999999, 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:27:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:27:37.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:27:37.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:27:37.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.649626215000001, 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:27:37.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:27:37.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:37.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:27:37.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:27:37.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:27:37.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:27:37.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:27:37.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:27:37.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:27:37.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:27:37.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:27:37.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:27:37.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:27:37.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:37.416 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:37.417 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.639663739, 2024-05-06T00:27:37.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:27:37.417 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.417 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.417 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:37.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:27:37.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:27:37.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:37.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:27:37.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:27:37.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:27:37.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:27:37.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.424 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.424 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.424 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:37.425 INFO:teuthology.orchestra.run.smithi186.stdout:2251 trunc from 0x4c3a00 to 0xb01a400 2024-05-06T00:27:37.426 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:37.426+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:37.427 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:37.526 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:37.526+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:37.560 INFO:teuthology.orchestra.run.smithi186.stdout:2252 punch from 0xb43400 to 0xb51200, (0xde00 bytes) 2024-05-06T00:27:37.563 INFO:teuthology.orchestra.run.smithi186.stdout:2253 punch from 0xa705600 to 0xa70ca00, (0x7400 bytes) 2024-05-06T00:27:37.564 INFO:teuthology.orchestra.run.smithi186.stdout:2254 punch from 0x171b400 to 0x1729000, (0xdc00 bytes) 2024-05-06T00:27:37.564 INFO:teuthology.orchestra.run.smithi186.stdout:2255 write 0x7ec3600 thru 0x7ec8409 (0x4e0a bytes) 2024-05-06T00:27:37.565 INFO:teuthology.orchestra.run.smithi186.stdout:2256 read 0x6894600 thru 0x6894cc6 (0x6c7 bytes) 2024-05-06T00:27:37.573 INFO:teuthology.orchestra.run.smithi186.stdout:2257 trunc from 0xb01a400 to 0xe3a5200 2024-05-06T00:27:37.613 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:37.613 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:37.628 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:37.626+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:37.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:37.650 INFO:teuthology.orchestra.run.smithi186.stdout:2258 write 0x5c80200 thru 0x5c8852e (0x832f bytes) 2024-05-06T00:27:37.672 INFO:teuthology.orchestra.run.smithi186.stdout:2261 trunc from 0xe3a5200 to 0x81e9400 2024-05-06T00:27:37.728 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:37.726+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:37.751 INFO:teuthology.orchestra.run.smithi186.stdout:2265 write 0xd01c800 thru 0xd02c4fd (0xfcfe bytes) 2024-05-06T00:27:37.752 INFO:teuthology.orchestra.run.smithi186.stdout:2267 punch from 0xcbc9400 to 0xcbce800, (0x5400 bytes) 2024-05-06T00:27:37.754 INFO:teuthology.orchestra.run.smithi186.stdout:2268 read 0x47af800 thru 0x47b4d2c (0x552d bytes) 2024-05-06T00:27:37.754 INFO:teuthology.orchestra.run.smithi186.stdout:2270 punch from 0xbf7d200 to 0xbf8ba00, (0xe800 bytes) 2024-05-06T00:27:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:37.754 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:37.754 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:37.754 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:37.755 INFO:teuthology.orchestra.run.smithi186.stdout:2271 punch from 0x3abc00 to 0x3afa00, (0x3e00 bytes) 2024-05-06T00:27:37.755 INFO:teuthology.orchestra.run.smithi186.stdout:2272 read 0xae18c00 thru 0xae1d0be (0x44bf bytes) 2024-05-06T00:27:37.756 INFO:teuthology.orchestra.run.smithi186.stdout:2274 read 0x5658800 thru 0x5667ce0 (0xf4e1 bytes) 2024-05-06T00:27:37.756 INFO:teuthology.orchestra.run.smithi186.stdout:2276 read 0x4249600 thru 0x4250a82 (0x7483 bytes) 2024-05-06T00:27:37.757 INFO:teuthology.orchestra.run.smithi186.stdout:2278 write 0x23e9e00 thru 0x23ecc96 (0x2e97 bytes) 2024-05-06T00:27:37.757 INFO:teuthology.orchestra.run.smithi186.stdout:2279 read 0xb2cae00 thru 0xb2d5628 (0xa829 bytes) 2024-05-06T00:27:37.757 INFO:teuthology.orchestra.run.smithi186.stdout:2280 read 0xbe8c400 thru 0xbe96437 (0xa038 bytes) 2024-05-06T00:27:37.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:37.777 INFO:teuthology.orchestra.run.smithi186.stdout:2284 trunc from 0xd02c600 to 0xf2ac00 2024-05-06T00:27:37.830 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:37.830+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:37.860 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:37.887 INFO:teuthology.orchestra.run.smithi186.stdout:2285 read 0x9f5800 thru 0x9facc9 (0x54ca bytes) 2024-05-06T00:27:37.888 INFO:teuthology.orchestra.run.smithi186.stdout:2286 read 0xf21c00 thru 0xf2ab01 (0x8f02 bytes) 2024-05-06T00:27:37.903 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:37.903 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:37.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:37.904 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:37.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.904 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:888 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3276800~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.904 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012201+0000", 2024-05-06T00:27:37.904 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.890766459, 2024-05-06T00:27:37.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044104892, 2024-05-06T00:27:37.904 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.904 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.905 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.905 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 888 2024-05-06T00:27:37.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.905 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012201+0000", 2024-05-06T00:27:37.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012201+0000", 2024-05-06T00:27:37.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012202+0000", 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3699999999999999e-07 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012205+0000", 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3100000000000001e-06 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012219+0000", 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3286e-05 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012228+0000", 2024-05-06T00:27:37.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.986e-06 2024-05-06T00:27:37.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.055801+0000", 2024-05-06T00:27:37.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043572781999999997 2024-05-06T00:27:37.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.055902+0000", 2024-05-06T00:27:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010126 2024-05-06T00:27:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056306+0000", 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040433100000000002 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:889 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3735552~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8907294060000002, 2024-05-06T00:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044456221999999997, 2024-05-06T00:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 889 2024-05-06T00:27:37.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.913 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:37.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:37.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012239+0000", 2024-05-06T00:27:37.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6000000000000003e-07 2024-05-06T00:27:37.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012243+0000", 2024-05-06T00:27:37.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5199999999999999e-06 2024-05-06T00:27:37.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012253+0000", 2024-05-06T00:27:37.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4620000000000002e-06 2024-05-06T00:27:37.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012260+0000", 2024-05-06T00:27:37.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4529999999999996e-06 2024-05-06T00:27:37.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056325+0000", 2024-05-06T00:27:37.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044065026 2024-05-06T00:27:37.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056386+0000", 2024-05-06T00:27:37.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0158999999999998e-05 2024-05-06T00:27:37.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056695+0000", 2024-05-06T00:27:37.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030894199999999998 2024-05-06T00:27:37.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.920 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:890 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3932160~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8906967669999997, 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045011909000000003, 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.921 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 890 2024-05-06T00:27:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012272+0000", 2024-05-06T00:27:37.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7000000000000003e-07 2024-05-06T00:27:37.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012276+0000", 2024-05-06T00:27:37.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:37.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012297+0000", 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0191999999999999e-05 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012309+0000", 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2354000000000001e-05 2024-05-06T00:27:37.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056761+0000", 2024-05-06T00:27:37.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044451630999999998 2024-05-06T00:27:37.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056861+0000", 2024-05-06T00:27:37.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100241 2024-05-06T00:27:37.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057283+0000", 2024-05-06T00:27:37.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042207799999999999 2024-05-06T00:27:37.927 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.927 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.928 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:891 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 4390912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.928 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:37.928 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8906360230000008, 2024-05-06T00:27:37.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045327649999999997, 2024-05-06T00:27:37.928 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 891 2024-05-06T00:27:37.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.929 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:37.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012333+0000", 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-07 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012338+0000", 2024-05-06T00:27:37.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5639999999999996e-06 2024-05-06T00:27:37.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012376+0000", 2024-05-06T00:27:37.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7608000000000003e-05 2024-05-06T00:27:37.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012401+0000", 2024-05-06T00:27:37.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4752999999999998e-05 2024-05-06T00:27:37.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057301+0000", 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044900714000000001 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057358+0000", 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6935000000000003e-05 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057659+0000", 2024-05-06T00:27:37.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030109599999999999 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:913 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 2457600~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8905409659999997, 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045143827999999997, 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 913 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:37.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012428+0000", 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7540000000000001e-06 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012446+0000", 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7492000000000001e-05 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012464+0000", 2024-05-06T00:27:37.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8267e-05 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012479+0000", 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4484999999999999e-05 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057352+0000", 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044873203 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057450+0000", 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7621999999999995e-05 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057571+0000", 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121005 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:914 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:37.939 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8904699180000009, 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045379520999999999, 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 914 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012499+0000", 2024-05-06T00:27:37.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1799999999999999e-06 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012504+0000", 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2599999999999996e-06 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012522+0000", 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7382999999999999e-05 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012539+0000", 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7849000000000001e-05 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057642+0000", 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045102439000000001 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057756+0000", 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114572 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057877+0000", 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120838 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:915 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.944 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8904061760000008, 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045460430000000003, 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 915 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012563+0000", 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0359999999999999e-06 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012568+0000", 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4500000000000003e-06 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012585+0000", 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7462999999999999e-05 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012602+0000", 2024-05-06T00:27:37.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6385999999999999e-05 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057895+0000", 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045292978999999997 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057949+0000", 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292999999999997e-05 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058022+0000", 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2823000000000004e-05 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:916 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3309568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8902719979999993, 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045666393, 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.950 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 916 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012697+0000", 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2100000000000001e-06 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012703+0000", 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8869999999999997e-06 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012720+0000", 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7073000000000001e-05 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012733+0000", 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3589e-05 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058086+0000", 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045352291000000003 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058205+0000", 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118794 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058362+0000", 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015754899999999999 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:917 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3768320~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8902144130000007, 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045746415999999998, 2024-05-06T00:27:37.955 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.956 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.956 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.958 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 917 2024-05-06T00:27:37.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012755+0000", 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3340000000000001e-06 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012761+0000", 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9359999999999996e-06 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012779+0000", 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8125999999999999e-05 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012792+0000", 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.327e-05 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058383+0000", 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045591499000000001 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058431+0000", 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7658999999999999e-05 2024-05-06T00:27:37.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058500+0000", 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8591999999999997e-05 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.962 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:918 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3964928~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8901571510000004, 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045976217999999999, 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 918 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012812+0000", 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2130000000000001e-06 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012819+0000", 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9630000000000001e-06 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012842+0000", 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3119999999999999e-05 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012856+0000", 2024-05-06T00:27:37.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3932e-05 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058570+0000", 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045713812999999999 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058662+0000", 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2893000000000007e-05 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058787+0000", 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000124284 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:919 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 4423680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8900932360000002, 2024-05-06T00:27:37.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046059062999999997, 2024-05-06T00:27:37.968 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.968 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.968 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 919 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012876+0000", 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.24e-06 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012881+0000", 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3460000000000001e-06 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012899+0000", 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7549000000000001e-05 2024-05-06T00:27:37.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012919+0000", 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0499000000000001e-05 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058807+0000", 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045887887000000002 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058862+0000", 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5325999999999998e-05 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058934+0000", 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1216000000000004e-05 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:920 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 4620288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.972 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8900303449999996, 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046322267, 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 920 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012939+0000", 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1799999999999999e-06 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012944+0000", 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4360000000000001e-06 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012962+0000", 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.783e-05 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012975+0000", 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3682999999999999e-05 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058988+0000", 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046012151000000001 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059133+0000", 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145253 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059260+0000", 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012673400000000001 2024-05-06T00:27:37.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:949 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 2457600~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8899735040000003, 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046356374999999998, 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.977 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 949 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012995+0000", 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013021+0000", 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5422000000000002e-05 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013040+0000", 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9012999999999999e-05 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013054+0000", 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4503e-05 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058955+0000", 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045900747999999998 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059006+0000", 2024-05-06T00:27:37.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1073999999999999e-05 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059351+0000", 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000344481 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:950 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8898929799999991, 2024-05-06T00:27:37.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046810959999999999, 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 950 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:37.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013076+0000", 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2100000000000001e-06 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013082+0000", 2024-05-06T00:27:37.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9569999999999999e-06 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013104+0000", 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2371999999999999e-05 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013119+0000", 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4769999999999999e-05 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059409+0000", 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046290169999999999 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059505+0000", 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5970000000000004e-05 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059886+0000", 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038051099999999999 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.985 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:951 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8898298180000008, 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047109395999999998, 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 951 2024-05-06T00:27:37.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013139+0000", 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.203e-06 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013144+0000", 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2329999999999997e-06 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013164+0000", 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0063000000000001e-05 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013178+0000", 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3906e-05 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059905+0000", 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046727168999999999 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059963+0000", 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7352999999999998e-05 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060247+0000", 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028446899999999998 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:952 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3309568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8897714570000002, 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047577908000000002, 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.990 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 952 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013197+0000", 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1829999999999999e-06 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013203+0000", 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5400000000000003e-06 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013224+0000", 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0993e-05 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013238+0000", 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3553000000000001e-05 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060310+0000", 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047072307000000001 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060409+0000", 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8793000000000001e-05 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060774+0000", 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036553900000000001 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.994 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:953 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3768320~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8897087550000009, 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047905053000000003, 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 953 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:37.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013260+0000", 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.096e-06 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:37.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013302+0000", 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1508999999999998e-05 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013325+0000", 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3926000000000001e-05 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013341+0000", 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5305999999999999e-05 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060795+0000", 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047453844000000002 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060860+0000", 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5749999999999999e-05 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061164+0000", 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030362199999999997 2024-05-06T00:27:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:954 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3964928~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.889581583, 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048325185, 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:37.999 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 954 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013387+0000", 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4300000000000001e-07 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013392+0000", 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5669999999999996e-06 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013405+0000", 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3873000000000001e-05 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013415+0000", 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2429999999999994e-06 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061227+0000", 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047811922999999999 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061327+0000", 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100046 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061711+0000", 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038458999999999998 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:955 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 4423680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8895416980000004, 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048668028000000002, 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:38.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 955 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013427+0000", 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013430+0000", 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013439+0000", 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4769999999999992e-06 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013446+0000", 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1690000000000003e-06 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061731+0000", 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048284322999999997 2024-05-06T00:27:38.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061789+0000", 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8251999999999997e-05 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062094+0000", 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030498999999999999 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.007 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:963 3.1a 3:5aa5ce21:::rbd_object_map.110c6c438b03:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8895113590000001, 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050888098, 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 963 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013457+0000", 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6400000000000001e-07 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013462+0000", 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6330000000000004e-06 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013472+0000", 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.009e-05 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013479+0000", 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4499999999999998e-06 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062158+0000", 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048678720000000002 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062268+0000", 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109966 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062780+0000", 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051193899999999999 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064025+0000", 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012450479999999999 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064186+0000", 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016146199999999999 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064285+0000", 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9079999999999993e-05 2024-05-06T00:27:38.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064344+0000", 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8946000000000001e-05 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:38.013 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:38.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:38.015 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:38.016 INFO:teuthology.orchestra.run.smithi186.stdout:2287 trunc from 0xf2ac00 to 0x8d87200 2024-05-06T00:27:38.016 INFO:teuthology.orchestra.run.smithi186.stdout:2288 read 0x3fd8e00 thru 0x3fd9df5 (0xff6 bytes) 2024-05-06T00:27:38.016 INFO:teuthology.orchestra.run.smithi186.stdout:2290 write 0xd0ba400 thru 0xd0c9979 (0xf57a bytes) 2024-05-06T00:27:38.016 INFO:teuthology.orchestra.run.smithi186.stdout:2292 read 0x8ece000 thru 0x8ed903e (0xb03f bytes) 2024-05-06T00:27:38.016 INFO:teuthology.orchestra.run.smithi186.stdout:2293 read 0x5f7ac00 thru 0x5f7ca9c (0x1e9d bytes) 2024-05-06T00:27:38.016 INFO:teuthology.orchestra.run.smithi186.stdout:2294 clone 17 order 23 su 262144 sc 9 2024-05-06T00:27:38.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:38.017 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:38.011+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:38.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:38.028 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:27:38.028 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:38.029 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-05-06T00:27:38.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:38.030 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:38.046 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:38.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:38.068 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:38.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:27:38.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:38.115 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:38.111+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:38.194 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:38.195 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:38.205 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone16 from 0xd0c9a00 (overlap 0x4c3a00) to 0x0 2024-05-06T00:27:38.210 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:38.253 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:38.251+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:38.318 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:38.315+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:38.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-06T00:27:38.342+0000 7fdfb36a9640 -1 mon.a@0(leader).osd e58 definitely_dead 0 2024-05-06T00:27:38.373 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:38.373 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:38.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:38.429 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:38.427+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:38.519 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:38.518+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:38.558 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:38.559 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:38.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:38.619 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:38.615+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:38.705 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:38.705 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:38.719 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:38.720 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:38.718+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:38.822 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:38.819+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:38.840 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:38.840 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:38.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:38.887 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-05-06T00:27:38.922 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:38.922+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:38.980 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:38.980 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:38.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:39.023 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:39.019+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:39.100 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.5. 2024-05-06T00:27:39.124 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:39.119+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:39.174 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:39.174 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:39.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:39.233 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:39.234+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:39.301 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:39.301 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:39.315 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:39.326 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:39.323+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:39.426 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:39.426+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:39.473 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:39.473 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:39.486 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:39.529 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:39.530+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:39.628 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:39.626+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:39.653 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:39.653 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:39.667 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:39.729 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:39.727+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:39.829 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:39.829 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:39.830 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:39.827+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:39.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:39.931 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:39.930+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:39.966 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:39.966 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:39.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:40.032 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:40.027+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:40.112 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:40.112 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:40.114 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:27:40.129 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:40.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:40.133 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:40.127+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:40.233 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:40.234+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:40.293 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:40.293 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:40.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:40.324 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:40.324 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:40.355 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:40.355 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:40.357 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:40.355+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:40.378 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:40.378 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:40.397 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:40.398 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:40.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:40.416 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:40.434 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:40.434 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.828987568000002, 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:40.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:40.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:40.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:40.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:40.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.556338869000001, 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:27:40.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:27:40.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:27:40.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.447 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.556303693, 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:27:40.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:27:40.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:27:40.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.556215013999999, 2024-05-06T00:27:40.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:27:40.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:27:40.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:27:40.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.556113070999999, 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:27:40.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:27:40.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.556003968999999, 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.465 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:27:40.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.555906109999999, 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:27:40.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.555841059999999, 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:40.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:27:40.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:27:40.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.555748414, 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:27:40.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:27:40.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.55567641, 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.489 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:27:40.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:27:40.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.796214924000001, 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:27:40.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:27:40.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:27:40.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.796195676, 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:27:40.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:27:40.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.796145555999999, 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:27:40.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.512 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.796124505000002, 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.513 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:27:40.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:27:40.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:27:40.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.796109185999999, 2024-05-06T00:27:40.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:40.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:27:40.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.796092303999998, 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:27:40.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:27:40.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.796073968000002, 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.532 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:40.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:27:40.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.796057965999999, 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:27:40.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:27:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.543 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.795971603000002, 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:27:40.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:27:40.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.786009127, 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:40.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:40.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:27:40.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:27:40.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:27:40.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:27:40.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:27:40.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.556 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:40.556 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:40.556 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:40.557 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:40.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:40.559 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:40.559 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:40.558+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:40.658 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:40.655+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:40.727 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:40.727 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:40.727 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:40.727 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:40.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:40.785 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:40.783+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:40.860 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:40.855+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:40.909 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:40.909 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:40.909 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:40.909 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:40.909 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:40.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:40.961 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:40.962+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:41.062 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:41.062+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:888 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3276800~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012201+0000", 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.07936099, 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044104892, 2024-05-06T00:27:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 888 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012201+0000", 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012201+0000", 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012202+0000", 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3699999999999999e-07 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012205+0000", 2024-05-06T00:27:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3100000000000001e-06 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012219+0000", 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3286e-05 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012228+0000", 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.986e-06 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.055801+0000", 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043572781999999997 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.055902+0000", 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010126 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056306+0000", 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040433100000000002 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:889 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3735552~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.079323937, 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044456221999999997, 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 889 2024-05-06T00:27:41.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012239+0000", 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6000000000000003e-07 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012243+0000", 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5199999999999999e-06 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012253+0000", 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4620000000000002e-06 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012260+0000", 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4529999999999996e-06 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056325+0000", 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044065026 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056386+0000", 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0158999999999998e-05 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056695+0000", 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030894199999999998 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:890 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3932160~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:41.102 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.079291297999999, 2024-05-06T00:27:41.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045011909000000003, 2024-05-06T00:27:41.103 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 890 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012272+0000", 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7000000000000003e-07 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012276+0000", 2024-05-06T00:27:41.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012297+0000", 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0191999999999999e-05 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012309+0000", 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2354000000000001e-05 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056761+0000", 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044451630999999998 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056861+0000", 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100241 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057283+0000", 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042207799999999999 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:891 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 4390912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.079230554, 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045327649999999997, 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 891 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012333+0000", 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-07 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012338+0000", 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5639999999999996e-06 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012376+0000", 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7608000000000003e-05 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012401+0000", 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4752999999999998e-05 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057301+0000", 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044900714000000001 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057358+0000", 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6935000000000003e-05 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057659+0000", 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030109599999999999 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:913 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 2457600~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:41.112 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.079135496999999, 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045143827999999997, 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 913 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012428+0000", 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7540000000000001e-06 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012446+0000", 2024-05-06T00:27:41.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7492000000000001e-05 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012464+0000", 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8267e-05 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012479+0000", 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4484999999999999e-05 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057352+0000", 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044873203 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057450+0000", 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7621999999999995e-05 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057571+0000", 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121005 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.116 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:914 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.079064449000001, 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045379520999999999, 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 914 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012499+0000", 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1799999999999999e-06 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012504+0000", 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2599999999999996e-06 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012522+0000", 2024-05-06T00:27:41.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7382999999999999e-05 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012539+0000", 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7849000000000001e-05 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057642+0000", 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045102439000000001 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057756+0000", 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114572 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057877+0000", 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120838 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:915 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.079000707000001, 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045460430000000003, 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 915 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012563+0000", 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0359999999999999e-06 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012568+0000", 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4500000000000003e-06 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012585+0000", 2024-05-06T00:27:41.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7462999999999999e-05 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012602+0000", 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6385999999999999e-05 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057895+0000", 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045292978999999997 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057949+0000", 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292999999999997e-05 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058022+0000", 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2823000000000004e-05 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:916 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3309568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078866529000001, 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045666393, 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 916 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012697+0000", 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2100000000000001e-06 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012703+0000", 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8869999999999997e-06 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012720+0000", 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7073000000000001e-05 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012733+0000", 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3589e-05 2024-05-06T00:27:41.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058086+0000", 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045352291000000003 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058205+0000", 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118794 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058362+0000", 2024-05-06T00:27:41.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015754899999999999 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:917 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3768320~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078808944, 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045746415999999998, 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.131 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 917 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:41.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012755+0000", 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3340000000000001e-06 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012761+0000", 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9359999999999996e-06 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012779+0000", 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8125999999999999e-05 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012792+0000", 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.327e-05 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058383+0000", 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045591499000000001 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058431+0000", 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7658999999999999e-05 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058500+0000", 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8591999999999997e-05 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:918 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3964928~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078751682, 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045976217999999999, 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 918 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.136 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012812+0000", 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2130000000000001e-06 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012819+0000", 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9630000000000001e-06 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012842+0000", 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3119999999999999e-05 2024-05-06T00:27:41.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012856+0000", 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3932e-05 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058570+0000", 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045713812999999999 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058662+0000", 2024-05-06T00:27:41.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2893000000000007e-05 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058787+0000", 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000124284 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:919 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 4423680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:41.140 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078687767, 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046059062999999997, 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 919 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:41.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012876+0000", 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.24e-06 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012881+0000", 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3460000000000001e-06 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012899+0000", 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7549000000000001e-05 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012919+0000", 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0499000000000001e-05 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058807+0000", 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045887887000000002 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058862+0000", 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5325999999999998e-05 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058934+0000", 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1216000000000004e-05 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:920 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 4620288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078624875999999, 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046322267, 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 920 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.145 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012939+0000", 2024-05-06T00:27:41.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1799999999999999e-06 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012944+0000", 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4360000000000001e-06 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012962+0000", 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.783e-05 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012975+0000", 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3682999999999999e-05 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058988+0000", 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046012151000000001 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059133+0000", 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145253 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059260+0000", 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012673400000000001 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:949 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 2457600~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078568035, 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046356374999999998, 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.149 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 949 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012995+0000", 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013021+0000", 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5422000000000002e-05 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013040+0000", 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9012999999999999e-05 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013054+0000", 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4503e-05 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058955+0000", 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045900747999999998 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059006+0000", 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1073999999999999e-05 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059351+0000", 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000344481 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.153 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:950 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078487511000001, 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046810959999999999, 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 950 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013076+0000", 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2100000000000001e-06 2024-05-06T00:27:41.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013082+0000", 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9569999999999999e-06 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013104+0000", 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2371999999999999e-05 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013119+0000", 2024-05-06T00:27:41.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4769999999999999e-05 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059409+0000", 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046290169999999999 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059505+0000", 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5970000000000004e-05 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059886+0000", 2024-05-06T00:27:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038051099999999999 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:951 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078424349000001, 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047109395999999998, 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 951 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013139+0000", 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.203e-06 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013144+0000", 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2329999999999997e-06 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013164+0000", 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0063000000000001e-05 2024-05-06T00:27:41.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013178+0000", 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3906e-05 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059905+0000", 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046727168999999999 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059963+0000", 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7352999999999998e-05 2024-05-06T00:27:41.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060247+0000", 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028446899999999998 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:952 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3309568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078365988, 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047577908000000002, 2024-05-06T00:27:41.162 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 952 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013197+0000", 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1829999999999999e-06 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013203+0000", 2024-05-06T00:27:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5400000000000003e-06 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013224+0000", 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0993e-05 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013238+0000", 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3553000000000001e-05 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060310+0000", 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047072307000000001 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060409+0000", 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8793000000000001e-05 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060774+0000", 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036553900000000001 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:953 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3768320~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078303286000001, 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047905053000000003, 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 953 2024-05-06T00:27:41.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013260+0000", 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.096e-06 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013302+0000", 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1508999999999998e-05 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013325+0000", 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3926000000000001e-05 2024-05-06T00:27:41.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013341+0000", 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5305999999999999e-05 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060795+0000", 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047453844000000002 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060860+0000", 2024-05-06T00:27:41.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5749999999999999e-05 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061164+0000", 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030362199999999997 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:954 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3964928~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:41.171 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078176114, 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048325185, 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 954 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:41.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013387+0000", 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4300000000000001e-07 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013392+0000", 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5669999999999996e-06 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013405+0000", 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3873000000000001e-05 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013415+0000", 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2429999999999994e-06 2024-05-06T00:27:41.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061227+0000", 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047811922999999999 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061327+0000", 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100046 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061711+0000", 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038458999999999998 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:955 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 4423680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.078136229, 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048668028000000002, 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 955 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013427+0000", 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013430+0000", 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013439+0000", 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4769999999999992e-06 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013446+0000", 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1690000000000003e-06 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061731+0000", 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048284322999999997 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061789+0000", 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8251999999999997e-05 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062094+0000", 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030498999999999999 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:963 3.1a 3:5aa5ce21:::rbd_object_map.110c6c438b03:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.07810589, 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050888098, 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.180 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 963 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013457+0000", 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6400000000000001e-07 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013462+0000", 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6330000000000004e-06 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013472+0000", 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.009e-05 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013479+0000", 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4499999999999998e-06 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062158+0000", 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048678720000000002 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062268+0000", 2024-05-06T00:27:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109966 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062780+0000", 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051193899999999999 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064025+0000", 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012450479999999999 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:41.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064186+0000", 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016146199999999999 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064285+0000", 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9079999999999993e-05 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064344+0000", 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8946000000000001e-05 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.186 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.186 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:41.188 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:27:41.188 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:41.189 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:41.186+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:41.289 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:41.283+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:41.339 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:41.339 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:41.339 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:41.339 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:41.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:41.405 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:41.403+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:41.489 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:41.487+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:41.543 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:41.543 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:41.543 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:41.543 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:41.543 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:41.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:41.601 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:41.599+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:41.691 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:41.690+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.268212971000001, 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:27:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:27:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.267410982000001, 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:27:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.267324528, 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.267251935000001, 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:27:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:27:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.267192084000001, 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:27:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:27:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.267116400999999, 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:27:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:27:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:27:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2967 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 159744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.259492413, 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026274390000000002, 2024-05-06T00:27:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2967 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467898+0000", 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.046e-06 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467917+0000", 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:27:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467937+0000", 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9613999999999999e-05 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467961+0000", 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.419e-05 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.493935+0000", 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025973922 2024-05-06T00:27:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494053+0000", 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011866099999999999 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494171+0000", 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011806099999999999 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.259410828, 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.259356620999998, 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:27:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.259304472, 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:27:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:27:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3040 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.257449149999999, 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026298018999999999, 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3040 2024-05-06T00:27:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469941+0000", 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0979999999999999e-06 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469946+0000", 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0170000000000004e-06 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469963+0000", 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6541999999999998e-05 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469977+0000", 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4015999999999999e-05 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495954+0000", 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025976902999999999 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496101+0000", 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147529 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496238+0000", 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136914 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.257346757000001, 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:27:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.257270417000001, 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:27:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.442132709999999, 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.442080269000002, 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.435060285999999, 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.435013574999999, 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:27:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.434683466999999, 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:27:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:27:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.434655544999998, 2024-05-06T00:27:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:27:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.423788655999999, 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:41.824 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:41.825 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:41.823+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:41.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:41.927 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:41.923+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:41.959 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4955.0:5833 3.c 3.8145d86c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.300380166, 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.30042626300000003, 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4955", 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5833 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660487+0000", 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0979999999999996e-06 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660495+0000", 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2989999999999997e-06 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660514+0000", 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8190999999999999e-05 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660526+0000", 2024-05-06T00:27:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2892e-05 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660667+0000", 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014097599999999999 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660686+0000", 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.889e-05 2024-05-06T00:27:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4955.0:5834 3.c 3.8145d86c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.300275295, 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.30043874199999998, 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:27:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4955", 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5834 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660586+0000", 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660595+0000", 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2969999999999992e-06 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660607+0000", 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1685000000000001e-05 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660615+0000", 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6610000000000006e-06 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660776+0000", 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016069399999999999 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:41.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660795+0000", 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9398e-05 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 2 2024-05-06T00:27:41.968 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:41.973 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:42.027 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:42.026+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:42.096 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:42.096 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:42.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:42.128 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:42.127+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:42.229 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:42.230+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:42.280 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.280 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.474193999999997, 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:42.286 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.474177797000003, 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2527 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch ping cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.118083847999998, 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16552214100000001, 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2527 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:42.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160471+0000", 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.206e-06 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160477+0000", 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7440000000000001e-06 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160491+0000", 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3739e-05 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160506+0000", 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.473e-05 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160712+0000", 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020643899999999999 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160736+0000", 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3676000000000002e-05 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325517+0000", 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16478073700000001 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325685+0000", 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016879300000000001 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325992+0000", 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030607699999999998 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:1976 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.12149286, 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050344819999999998, 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1976 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157061+0000", 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3200000000000005e-07 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157064+0000", 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.022e-06 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157091+0000", 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6860999999999999e-05 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157101+0000", 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0293e-05 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207258+0000", 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050156559000000003 2024-05-06T00:27:42.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207333+0000", 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5426000000000006e-05 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207405+0000", 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2026999999999993e-05 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:1977 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 874496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1214415720000002, 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050463471000000003, 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1977 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.302 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5400000000000001e-07 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157116+0000", 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3720000000000001e-06 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157125+0000", 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5310000000000007e-06 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157132+0000", 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.815e-06 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207435+0000", 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050303193000000003 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207484+0000", 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8646000000000001e-05 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207575+0000", 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1260000000000004e-05 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2109 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 465920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1214123259999997, 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050607512, 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2109 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.89e-07 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157144+0000", 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0340000000000001e-06 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157153+0000", 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2409999999999995e-06 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157159+0000", 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6719999999999998e-06 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207604+0000", 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050444298999999998 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207674+0000", 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0636000000000006e-05 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207748+0000", 2024-05-06T00:27:42.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4140999999999994e-05 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2110 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1213857020000004, 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050757732999999999, 2024-05-06T00:27:42.311 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2110 2024-05-06T00:27:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:42.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:42.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157175+0000", 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5100000000000004e-06 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157183+0000", 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3429999999999996e-06 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157189+0000", 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1140000000000002e-06 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207777+0000", 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050587886999999998 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207851+0000", 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4368000000000004e-05 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207925+0000", 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4046000000000002e-05 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2111 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 486400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1213531190000001, 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050893031999999998, 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2111 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157201+0000", 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7899999999999998e-07 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157204+0000", 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0309999999999999e-06 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157212+0000", 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6719999999999993e-06 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157219+0000", 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1750000000000002e-06 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207953+0000", 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050734855000000002 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208022+0000", 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8468999999999998e-05 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208093+0000", 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1250999999999999e-05 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2112 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 496640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.322 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1213261169999997, 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051032028, 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2112 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157228+0000", 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8999999999999997e-07 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157230+0000", 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8049999999999998e-06 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157239+0000", 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2889999999999998e-06 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157245+0000", 2024-05-06T00:27:42.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3389999999999998e-06 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208121+0000", 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050875709999999998 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208189+0000", 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8711e-05 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208259+0000", 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9684000000000006e-05 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2113 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:42.327 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1212999129999996, 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051203918000000001, 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2113 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157655+0000", 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040158000000000001 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157661+0000", 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5840000000000002e-06 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157675+0000", 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4354e-05 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157684+0000", 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1859999999999998e-06 2024-05-06T00:27:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208288+0000", 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050603883000000002 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208357+0000", 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8884e-05 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208457+0000", 2024-05-06T00:27:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100447 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2114 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 517120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1208525119999999, 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050919006000000003, 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2114 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158519+0000", 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00081808800000000002 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158524+0000", 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.558e-06 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158538+0000", 2024-05-06T00:27:42.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3703000000000001e-05 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158548+0000", 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0180000000000001e-05 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208486+0000", 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049937466 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208551+0000", 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-05 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208620+0000", 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8281000000000006e-05 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2115 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 527360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158564+0000", 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1199891429999997, 2024-05-06T00:27:42.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050220562000000003, 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2115 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158564+0000", 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158564+0000", 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158565+0000", 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6300000000000004e-07 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158569+0000", 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158579+0000", 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0137e-05 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158587+0000", 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8660000000000006e-06 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208649+0000", 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050062500000000003 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208713+0000", 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4281000000000003e-05 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208785+0000", 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1375e-05 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2116 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 537600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1199559209999999, 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050356630999999999, 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2116 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158598+0000", 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7700000000000004e-07 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158602+0000", 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4300000000000002e-06 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158610+0000", 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6510000000000007e-06 2024-05-06T00:27:42.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158616+0000", 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1330000000000004e-06 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208813+0000", 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050196527999999997 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208881+0000", 2024-05-06T00:27:42.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7718000000000005e-05 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208954+0000", 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3393999999999996e-05 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2117 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 547840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1199268599999996, 2024-05-06T00:27:42.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050509317999999997, 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2117 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:42.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158627+0000", 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3200000000000005e-07 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158656+0000", 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8989999999999999e-05 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158666+0000", 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0093e-05 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158673+0000", 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3170000000000004e-06 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208987+0000", 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050313331000000003 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209059+0000", 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2151999999999996e-05 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209136+0000", 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6803000000000003e-05 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2118 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 558080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1198679580000004, 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.05062726, 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2118 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158686+0000", 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158689+0000", 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0929999999999999e-06 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158697+0000", 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8560000000000007e-06 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158704+0000", 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8970000000000002e-06 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209167+0000", 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050462662999999998 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209239+0000", 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2700999999999994e-05 2024-05-06T00:27:42.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209313+0000", 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3301999999999997e-05 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2119 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 568320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1198396109999997, 2024-05-06T00:27:42.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050765355999999998, 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2119 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7899999999999998e-07 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158717+0000", 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1140000000000002e-06 2024-05-06T00:27:42.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158726+0000", 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2470000000000008e-06 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158735+0000", 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0429999999999996e-06 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209342+0000", 2024-05-06T00:27:42.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050607632999999999 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209415+0000", 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2675999999999993e-05 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209479+0000", 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4064000000000002e-05 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2120 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 578560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1198085500000001, 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050922111999999999, 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2120 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158770+0000", 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4830000000000001e-05 2024-05-06T00:27:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158774+0000", 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.656e-06 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158784+0000", 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3729999999999999e-06 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158791+0000", 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1860000000000003e-06 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209512+0000", 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050721016000000001 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209593+0000", 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1629000000000001e-05 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209667+0000", 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3422000000000003e-05 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2121 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 588800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1197515390000001, 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051033950000000002, 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2121 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162924+0000", 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041224440000000003 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162931+0000", 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7909999999999999e-06 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162943+0000", 2024-05-06T00:27:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2009e-05 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162952+0000", 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1160000000000005e-06 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209698+0000", 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046745791000000002 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209770+0000", 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2230000000000005e-05 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209836+0000", 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5568999999999995e-05 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2148 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 865280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.163848+0000", 2024-05-06T00:27:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1147048599999998, 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050291534999999998, 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2148 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163848+0000", 2024-05-06T00:27:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163848+0000", 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163849+0000", 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0699999999999997e-07 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163851+0000", 2024-05-06T00:27:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.543e-06 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163859+0000", 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163865+0000", 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214024+0000", 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050158871000000001 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214061+0000", 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6829000000000003e-05 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214140+0000", 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8770999999999995e-05 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2149 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 875520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1146798440000003, 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050369018000000002, 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2149 2024-05-06T00:27:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163874+0000", 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9699999999999996e-07 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163877+0000", 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.497e-06 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163885+0000", 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0539999999999995e-06 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163891+0000", 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2639999999999997e-06 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214165+0000", 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050274513999999999 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214202+0000", 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6390000000000002e-05 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214242+0000", 2024-05-06T00:27:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0701999999999998e-05 2024-05-06T00:27:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:42.376 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:42.377 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:42.378 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:42.375+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:42.478 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:42.479+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:42.546 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:42.547 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:42.547 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:42.547 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:42.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:42.561 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:42.579 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:42.575+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:42.680 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:42.678+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:42.705 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:42.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:42.705 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:42.705 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:42.705 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:42.723 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:42.723 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:42.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:42.744 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:42.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:42.766 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:42.782 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:42.779+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:42.789 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:42.789 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:42.814 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:42.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:42.841 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:42.883 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:42.879+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:42.887 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:42.887 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:42.903 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:42.983 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:42.979+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:43.074 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:43.075 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:43.084 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:43.079+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:43.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:43.185 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:43.183+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:43.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:27:43.210 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:43.210 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:43.225 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:43.286 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:43.283+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:43.329 INFO:teuthology.orchestra.run.smithi186.stdout:2297 trunc from 0xd0c9a00 to 0xd4d2800 2024-05-06T00:27:43.351 INFO:teuthology.orchestra.run.smithi186.stdout:2298 write 0x2585600 thru 0x25865f1 (0xff2 bytes) 2024-05-06T00:27:43.354 INFO:teuthology.orchestra.run.smithi186.stdout:2299 read 0x19e3200 thru 0x19e5c67 (0x2a68 bytes) 2024-05-06T00:27:43.354 INFO:teuthology.orchestra.run.smithi186.stdout:2300 punch from 0x5605e00 to 0x560c400, (0x6600 bytes) 2024-05-06T00:27:43.370 INFO:teuthology.orchestra.run.smithi186.stdout:2302 read 0x2eeaa00 thru 0x2ef7fb9 (0xd5ba bytes) 2024-05-06T00:27:43.371 INFO:teuthology.orchestra.run.smithi186.stdout:2305 read 0x80c3c00 thru 0x80ce6be (0xaabf bytes) 2024-05-06T00:27:43.371 INFO:teuthology.orchestra.run.smithi186.stdout:2309 punch from 0xc607600 to 0xc613200, (0xbc00 bytes) 2024-05-06T00:27:43.373 INFO:teuthology.orchestra.run.smithi186.stdout:2312 read 0x6318200 thru 0x631bf2b (0x3d2c bytes) 2024-05-06T00:27:43.374 INFO:teuthology.orchestra.run.smithi186.stdout:2314 read 0x3357000 thru 0x3358177 (0x1178 bytes) 2024-05-06T00:27:43.374 INFO:teuthology.orchestra.run.smithi186.stdout:2315 read 0x7144200 thru 0x71443c4 (0x1c5 bytes) 2024-05-06T00:27:43.374 INFO:teuthology.orchestra.run.smithi186.stdout:2317 read 0x7bc6600 thru 0x7bca3ac (0x3dad bytes) 2024-05-06T00:27:43.386 INFO:teuthology.orchestra.run.smithi186.stdout:2318 trunc from 0xd4d2800 to 0xb136400 2024-05-06T00:27:43.387 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:43.386+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:43.410 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:43.410 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:43.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:43.443 INFO:teuthology.orchestra.run.smithi186.stdout:2319 punch from 0x7b4d000 to 0x7b53800, (0x6800 bytes) 2024-05-06T00:27:43.449 INFO:teuthology.orchestra.run.smithi186.stdout:2320 read 0x1a1e000 thru 0x1a2b6ed (0xd6ee bytes) 2024-05-06T00:27:43.449 INFO:teuthology.orchestra.run.smithi186.stdout:2321 write 0x7d3f400 thru 0x7d49bef (0xa7f0 bytes) 2024-05-06T00:27:43.482 INFO:teuthology.orchestra.run.smithi186.stdout:2323 write 0xd76bc00 thru 0xd776b90 (0xaf91 bytes) 2024-05-06T00:27:43.483 INFO:teuthology.orchestra.run.smithi186.stdout:2329 write 0x32d9e00 thru 0x32e00eb (0x62ec bytes) 2024-05-06T00:27:43.483 INFO:teuthology.orchestra.run.smithi186.stdout:2331 punch from 0xb3e9200 to 0xb3f0e00, (0x7c00 bytes) 2024-05-06T00:27:43.487 INFO:teuthology.orchestra.run.smithi186.stdout:2332 read 0x1591200 thru 0x1599f0b (0x8d0c bytes) 2024-05-06T00:27:43.487 INFO:teuthology.orchestra.run.smithi186.stdout:2333 write 0x821a00 thru 0x82dc0c (0xc20d bytes) 2024-05-06T00:27:43.487 INFO:teuthology.orchestra.run.smithi186.stdout:2336 punch from 0xb7de600 to 0xb7e4c00, (0x6600 bytes) 2024-05-06T00:27:43.488 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:43.483+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:43.488 INFO:teuthology.orchestra.run.smithi186.stdout:2339 write 0x3b24400 thru 0x3b288df (0x44e0 bytes) 2024-05-06T00:27:43.488 INFO:teuthology.orchestra.run.smithi186.stdout:2340 read 0x68e0800 thru 0x68e4701 (0x3f02 bytes) 2024-05-06T00:27:43.490 INFO:teuthology.orchestra.run.smithi186.stdout:2341 read 0x6048400 thru 0x604bcb2 (0x38b3 bytes) 2024-05-06T00:27:43.491 INFO:teuthology.orchestra.run.smithi186.stdout:2343 read 0x8eaf600 thru 0x8eb3f76 (0x4977 bytes) 2024-05-06T00:27:43.556 INFO:teuthology.orchestra.run.smithi186.stdout:2345 trunc from 0xd776c00 to 0x91ca00 2024-05-06T00:27:43.566 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:43.567 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:43.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:43.588 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:43.586+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:27:43.680 INFO:teuthology.orchestra.run.smithi186.stdout:2350 trunc from 0x91ca00 to 0xd4fe400 2024-05-06T00:27:43.693 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:43.694+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:43.694 INFO:teuthology.orchestra.run.smithi186.stdout:2351 write 0x69eba00 thru 0x69f90a2 (0xd6a3 bytes) 2024-05-06T00:27:43.696 INFO:teuthology.orchestra.run.smithi186.stdout:2352 write 0x86a9a00 thru 0x86b0bac (0x71ad bytes) 2024-05-06T00:27:43.696 INFO:teuthology.orchestra.run.smithi186.stdout:2355 punch from 0x6151800 to 0x6157c00, (0x6400 bytes) 2024-05-06T00:27:43.696 INFO:teuthology.orchestra.run.smithi186.stdout:2356 write 0x2357400 thru 0x235faff (0x8700 bytes) 2024-05-06T00:27:43.698 INFO:teuthology.orchestra.run.smithi186.stdout:2357 write 0x5077e00 thru 0x507fbfa (0x7dfb bytes) 2024-05-06T00:27:43.707 INFO:teuthology.orchestra.run.smithi186.stdout:2360 trunc from 0xd4fe400 to 0x8cfb200 2024-05-06T00:27:43.738 INFO:teuthology.orchestra.run.smithi186.stdout:2361 read 0x9bf800 thru 0x9cc566 (0xcd67 bytes) 2024-05-06T00:27:43.742 INFO:teuthology.orchestra.run.smithi186.stdout:2365 write 0x1a84400 thru 0x1a876c7 (0x32c8 bytes) 2024-05-06T00:27:43.743 INFO:teuthology.orchestra.run.smithi186.stdout:2366 write 0x6ea5800 thru 0x6eb1ec8 (0xc6c9 bytes) 2024-05-06T00:27:43.743 INFO:teuthology.orchestra.run.smithi186.stdout:2367 punch from 0x5fdd400 to 0x5feb800, (0xe400 bytes) 2024-05-06T00:27:43.744 INFO:teuthology.orchestra.run.smithi186.stdout:2368 read 0x2a2a600 thru 0x2a2d874 (0x3275 bytes) 2024-05-06T00:27:43.745 INFO:teuthology.orchestra.run.smithi186.stdout:2369 read 0x7a8000 thru 0x7adced (0x5cee bytes) 2024-05-06T00:27:43.746 INFO:teuthology.orchestra.run.smithi186.stdout:2370 write 0x1cd8600 thru 0x1ce1724 (0x9125 bytes) 2024-05-06T00:27:43.757 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:43.757 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:43.760 INFO:teuthology.orchestra.run.smithi186.stdout:2371 trunc from 0x8cfb200 to 0x9a8000 2024-05-06T00:27:43.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:43.790 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:43.790+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:43.804 INFO:teuthology.orchestra.run.smithi186.stdout:2373 read 0x375c00 thru 0x3810af (0xb4b0 bytes) 2024-05-06T00:27:43.807 INFO:teuthology.orchestra.run.smithi186.stdout:2376 punch from 0x851400 to 0x85c000, (0xac00 bytes) 2024-05-06T00:27:43.808 INFO:teuthology.orchestra.run.smithi186.stdout:2377 clone 18 order 22 su 524288 sc 10 2024-05-06T00:27:43.891 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:43.890+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:43.893 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:43.894 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:43.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:43.992 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:43.990+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:44.027 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:44.028 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:44.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:44.093 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:44.094+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:44.127 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:44.127 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:27:44.128 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:27:44.163 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:44.163 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:44.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:44.194 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:44.194+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:44.295 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:44.294+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:44.302 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:44.302 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:44.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:44.396 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:44.391+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:44.439 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:44.439 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:44.440 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:27:44.441 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:27:44.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:44.457 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-05-06T00:27:44.412347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493677357,"num_objects":847,"num_object_clones":149,"num_object_copies":1694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":847,"num_whiteouts":479,"num_read":80506,"num_read_kb":341956,"num_write":7449,"num_write_kb":91300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":5173094,"num_keys_recovered":138,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6885,"ondisk_log_size":6885,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":237576,"kb_used_data":48388,"kb_used_omap":7,"kb_used_meta":189112,"kb_avail":471621624,"statfs":{"total":483183820800,"available":482940542976,"internally_reserved":0,"allocated":49549312,"data_stored":73103947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7973,"internal_metadata":193650907},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3990308,"num_objects":69,"num_object_clones":38,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":40,"num_read":5085,"num_read_kb":14990,"num_write":489,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":462,"num_bytes_recovered":5173059,"num_keys_recovered":126,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001043"},"pg_stats":[{"pgid":"3.1f","version":"57'181","reported_seq":3355,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.608349+0000","last_change":"2024-05-06T00:26:45.712771+0000","last_active":"2024-05-06T00:27:41.608349+0000","last_peered":"2024-05-06T00:27:41.608349+0000","last_clean":"2024-05-06T00:27:41.608349+0000","last_became_active":"2024-05-06T00:26:43.694494+0000","last_became_peered":"2024-05-06T00:26:43.694494+0000","last_unstale":"2024-05-06T00:27:41.608349+0000","last_undegraded":"2024-05-06T00:27:41.608349+0000","last_fullsized":"2024-05-06T00:27:41.608349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:21:51.684875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":9127047,"num_objects":26,"num_object_clones":3,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":14,"num_read":3422,"num_read_kb":25621,"num_write":218,"num_write_kb":821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"58'123","reported_seq":2008,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.605052+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:27:41.605052+0000","last_peered":"2024-05-06T00:27:41.605052+0000","last_clean":"2024-05-06T00:27:41.605052+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:27:41.605052+0000","last_undegraded":"2024-05-06T00:27:41.605052+0000","last_fullsized":"2024-05-06T00:27:41.605052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:17:11.749653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":41603599,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":15,"num_read":1843,"num_read_kb":22976,"num_write":110,"num_write_kb":12648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"58'105","reported_seq":906,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.604596+0000","last_change":"2024-05-06T00:27:41.604596+0000","last_active":"2024-05-06T00:27:41.604596+0000","last_peered":"2024-05-06T00:27:41.296897+0000","last_clean":"2024-05-06T00:27:38.102195+0000","last_became_active":"2024-05-06T00:27:41.126599+0000","last_became_peered":"2024-05-06T00:27:41.126599+0000","last_unstale":"2024-05-06T00:27:41.604596+0000","last_undegraded":"2024-05-06T00:27:41.604596+0000","last_fullsized":"2024-05-06T00:27:41.604596+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":9088000,"num_objects":24,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":17,"num_read":620,"num_read_kb":7225,"num_write":88,"num_write_kb":492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":69632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"58'418","reported_seq":5409,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.604286+0000","last_change":"2024-05-06T00:27:41.604286+0000","last_active":"2024-05-06T00:27:41.604286+0000","last_peered":"2024-05-06T00:27:41.435280+0000","last_clean":"2024-05-06T00:27:38.178877+0000","last_became_active":"2024-05-06T00:27:41.136822+0000","last_became_peered":"2024-05-06T00:27:41.136822+0000","last_unstale":"2024-05-06T00:27:41.604286+0000","last_undegraded":"2024-05-06T00:27:41.604286+0000","last_fullsized":"2024-05-06T00:27:41.604286+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":13794357,"num_objects":31,"num_object_clones":7,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":15,"num_read":6952,"num_read_kb":22308,"num_write":394,"num_write_kb":3122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":510005,"num_keys_recovered":60,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"58'451","reported_seq":4571,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.607346+0000","last_change":"2024-05-06T00:26:45.713563+0000","last_active":"2024-05-06T00:27:41.607346+0000","last_peered":"2024-05-06T00:27:41.607346+0000","last_clean":"2024-05-06T00:27:41.607346+0000","last_became_active":"2024-05-06T00:26:43.707253+0000","last_became_peered":"2024-05-06T00:26:43.707253+0000","last_unstale":"2024-05-06T00:27:41.607346+0000","last_undegraded":"2024-05-06T00:27:41.607346+0000","last_fullsized":"2024-05-06T00:27:41.607346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:46:20.136112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":39589962,"num_objects":49,"num_object_clones":14,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":24,"num_read":6084,"num_read_kb":19986,"num_write":411,"num_write_kb":3410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"57'214","reported_seq":3155,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.603807+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:27:41.603807+0000","last_peered":"2024-05-06T00:27:41.603807+0000","last_clean":"2024-05-06T00:27:41.603807+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:27:41.603807+0000","last_undegraded":"2024-05-06T00:27:41.603807+0000","last_fullsized":"2024-05-06T00:27:41.603807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:37:26.811168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":22871721,"num_objects":31,"num_object_clones":4,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":14,"num_read":3681,"num_read_kb":18252,"num_write":258,"num_write_kb":10645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"58'152","reported_seq":1482,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.609586+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:27:41.609586+0000","last_peered":"2024-05-06T00:27:41.609586+0000","last_clean":"2024-05-06T00:27:41.609586+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:27:41.609586+0000","last_undegraded":"2024-05-06T00:27:41.609586+0000","last_fullsized":"2024-05-06T00:27:41.609586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:53:27.812739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":15043140,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":8,"num_read":1946,"num_read_kb":8063,"num_write":122,"num_write_kb":2753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"58'89","reported_seq":879,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.604603+0000","last_change":"2024-05-06T00:26:45.713666+0000","last_active":"2024-05-06T00:27:41.604603+0000","last_peered":"2024-05-06T00:27:41.604603+0000","last_clean":"2024-05-06T00:27:41.604603+0000","last_became_active":"2024-05-06T00:26:43.688284+0000","last_became_peered":"2024-05-06T00:26:43.688284+0000","last_unstale":"2024-05-06T00:27:41.604603+0000","last_undegraded":"2024-05-06T00:27:41.604603+0000","last_fullsized":"2024-05-06T00:27:41.604603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:54:05.588235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":3420160,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":10,"num_read":689,"num_read_kb":6163,"num_write":77,"num_write_kb":507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"58'139","reported_seq":932,"reported_epoch":61,"state":"peering","last_fresh":"2024-05-06T00:27:41.434212+0000","last_change":"2024-05-06T00:27:41.607137+0000","last_active":"2024-05-06T00:27:41.607137+0000","last_peered":"2024-05-06T00:27:41.434212+0000","last_clean":"2024-05-06T00:27:38.226218+0000","last_became_active":"2024-05-06T00:27:41.127739+0000","last_became_peered":"2024-05-06T00:27:41.127739+0000","last_unstale":"2024-05-06T00:27:41.434212+0000","last_undegraded":"2024-05-06T00:27:41.434212+0000","last_fullsized":"2024-05-06T00:27:41.116794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":35174954,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":16,"num_read":686,"num_read_kb":4616,"num_write":123,"num_write_kb":2999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":276010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"58'106","reported_seq":1184,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.224295+0000","last_change":"2024-05-06T00:27:42.501961+0000","last_active":"2024-05-06T00:27:42.501961+0000","last_peered":"2024-05-06T00:27:41.224295+0000","last_clean":"2024-05-06T00:27:38.171831+0000","last_became_active":"2024-05-06T00:27:41.119011+0000","last_became_peered":"2024-05-06T00:27:41.119011+0000","last_unstale":"2024-05-06T00:27:41.224295+0000","last_undegraded":"2024-05-06T00:27:41.224295+0000","last_fullsized":"2024-05-06T00:27:41.110066+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":37438023,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":15,"num_read":1042,"num_read_kb":10239,"num_write":98,"num_write_kb":594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":240711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"58'106","reported_seq":940,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.603651+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:27:41.603651+0000","last_peered":"2024-05-06T00:27:41.603651+0000","last_clean":"2024-05-06T00:27:41.603651+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:27:41.603651+0000","last_undegraded":"2024-05-06T00:27:41.603651+0000","last_fullsized":"2024-05-06T00:27:41.603651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:02:13.418052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6373888,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":17,"num_read":790,"num_read_kb":3466,"num_write":93,"num_write_kb":11946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"57'101","reported_seq":1071,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.603639+0000","last_change":"2024-05-06T00:27:21.967099+0000","last_active":"2024-05-06T00:27:41.603639+0000","last_peered":"2024-05-06T00:27:41.603639+0000","last_clean":"2024-05-06T00:27:41.603639+0000","last_became_active":"2024-05-06T00:27:21.964399+0000","last_became_peered":"2024-05-06T00:27:21.964399+0000","last_unstale":"2024-05-06T00:27:41.603639+0000","last_undegraded":"2024-05-06T00:27:41.603639+0000","last_fullsized":"2024-05-06T00:27:41.603639+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:01:34.492231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":15798800,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":13,"num_read":881,"num_read_kb":8275,"num_write":80,"num_write_kb":517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.603860+0000","last_change":"2024-05-06T00:27:21.965066+0000","last_active":"2024-05-06T00:27:41.603860+0000","last_peered":"2024-05-06T00:27:41.603860+0000","last_clean":"2024-05-06T00:27:41.603860+0000","last_became_active":"2024-05-06T00:27:21.964272+0000","last_became_peered":"2024-05-06T00:27:21.964272+0000","last_unstale":"2024-05-06T00:27:41.603860+0000","last_undegraded":"2024-05-06T00:27:41.603860+0000","last_fullsized":"2024-05-06T00:27:41.603860+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:06.226485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'507","reported_seq":3765,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.611300+0000","last_change":"2024-05-06T00:27:01.825846+0000","last_active":"2024-05-06T00:27:41.611300+0000","last_peered":"2024-05-06T00:27:41.611300+0000","last_clean":"2024-05-06T00:27:41.611300+0000","last_became_active":"2024-05-06T00:27:01.825382+0000","last_became_peered":"2024-05-06T00:27:01.825382+0000","last_unstale":"2024-05-06T00:27:41.611300+0000","last_undegraded":"2024-05-06T00:27:41.611300+0000","last_fullsized":"2024-05-06T00:27:41.611300+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:15:11.567357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":4247122,"num_objects":24,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":12,"num_read":6585,"num_read_kb":10803,"num_write":523,"num_write_kb":2114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":35,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.609806+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:27:41.609806+0000","last_peered":"2024-05-06T00:27:41.609806+0000","last_clean":"2024-05-06T00:27:41.609806+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:27:41.609806+0000","last_undegraded":"2024-05-06T00:27:41.609806+0000","last_fullsized":"2024-05-06T00:27:41.609806+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"58'329","reported_seq":1320,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.605326+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:27:41.605326+0000","last_peered":"2024-05-06T00:27:41.605326+0000","last_clean":"2024-05-06T00:27:41.605326+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:27:41.605326+0000","last_undegraded":"2024-05-06T00:27:41.605326+0000","last_fullsized":"2024-05-06T00:27:41.605326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:07:52.825348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":2227237,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":10,"num_read":2657,"num_read_kb":3776,"num_write":418,"num_write_kb":877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.607458+0000","last_change":"2024-05-06T00:27:21.962747+0000","last_active":"2024-05-06T00:27:41.607458+0000","last_peered":"2024-05-06T00:27:41.607458+0000","last_clean":"2024-05-06T00:27:41.607458+0000","last_became_active":"2024-05-06T00:27:21.962380+0000","last_became_peered":"2024-05-06T00:27:21.962380+0000","last_unstale":"2024-05-06T00:27:41.607458+0000","last_undegraded":"2024-05-06T00:27:41.607458+0000","last_fullsized":"2024-05-06T00:27:41.607458+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:56.819600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"58'336","reported_seq":1550,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.609530+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:27:41.609530+0000","last_peered":"2024-05-06T00:27:41.609530+0000","last_clean":"2024-05-06T00:27:41.609530+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:27:41.609530+0000","last_undegraded":"2024-05-06T00:27:41.609530+0000","last_fullsized":"2024-05-06T00:27:41.609530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:32:25.906829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":21137477,"num_objects":36,"num_object_clones":8,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":22,"num_read":2001,"num_read_kb":12006,"num_write":448,"num_write_kb":2014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.609737+0000","last_change":"2024-05-06T00:26:35.646598+0000","last_active":"2024-05-06T00:27:41.609737+0000","last_peered":"2024-05-06T00:27:41.609737+0000","last_clean":"2024-05-06T00:27:41.609737+0000","last_became_active":"2024-05-06T00:26:35.646239+0000","last_became_peered":"2024-05-06T00:26:35.646239+0000","last_unstale":"2024-05-06T00:27:41.609737+0000","last_undegraded":"2024-05-06T00:27:41.609737+0000","last_fullsized":"2024-05-06T00:27:41.609737+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:53:27.613539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"57'111","reported_seq":915,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.606336+0000","last_change":"2024-05-06T00:27:41.606336+0000","last_active":"2024-05-06T00:27:41.606336+0000","last_peered":"2024-05-06T00:27:41.499108+0000","last_clean":"2024-05-06T00:27:37.835899+0000","last_became_active":"2024-05-06T00:27:41.122698+0000","last_became_peered":"2024-05-06T00:27:41.122698+0000","last_unstale":"2024-05-06T00:27:41.606336+0000","last_undegraded":"2024-05-06T00:27:41.606336+0000","last_fullsized":"2024-05-06T00:27:41.606336+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":2220064,"num_objects":19,"num_object_clones":2,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":11,"num_read":537,"num_read_kb":4851,"num_write":99,"num_write_kb":633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":51232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"58'77","reported_seq":866,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.603808+0000","last_change":"2024-05-06T00:26:45.720211+0000","last_active":"2024-05-06T00:27:41.603808+0000","last_peered":"2024-05-06T00:27:41.603808+0000","last_clean":"2024-05-06T00:27:41.603808+0000","last_became_active":"2024-05-06T00:26:43.705235+0000","last_became_peered":"2024-05-06T00:26:43.705235+0000","last_unstale":"2024-05-06T00:27:41.603808+0000","last_undegraded":"2024-05-06T00:27:41.603808+0000","last_fullsized":"2024-05-06T00:27:41.603808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:53:50.810279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":5262373,"num_objects":20,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":11,"num_read":695,"num_read_kb":6053,"num_write":66,"num_write_kb":455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.608426+0000","last_change":"2024-05-06T00:27:40.122950+0000","last_active":"2024-05-06T00:27:41.608426+0000","last_peered":"2024-05-06T00:27:41.608426+0000","last_clean":"2024-05-06T00:27:41.608426+0000","last_became_active":"2024-05-06T00:27:40.122044+0000","last_became_peered":"2024-05-06T00:27:40.122044+0000","last_unstale":"2024-05-06T00:27:41.608426+0000","last_undegraded":"2024-05-06T00:27:41.608426+0000","last_fullsized":"2024-05-06T00:27:41.608426+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:35:57.600975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'351","reported_seq":2773,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.609080+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:27:41.609080+0000","last_peered":"2024-05-06T00:27:41.609080+0000","last_clean":"2024-05-06T00:27:41.609080+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:27:41.609080+0000","last_undegraded":"2024-05-06T00:27:41.609080+0000","last_fullsized":"2024-05-06T00:27:41.609080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:17:00.748050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":9586822,"num_objects":27,"num_object_clones":2,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":13,"num_read":3640,"num_read_kb":13221,"num_write":532,"num_write_kb":2879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.604739+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:27:41.604739+0000","last_peered":"2024-05-06T00:27:41.604739+0000","last_clean":"2024-05-06T00:27:41.604739+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:27:41.604739+0000","last_undegraded":"2024-05-06T00:27:41.604739+0000","last_fullsized":"2024-05-06T00:27:41.604739+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"57'91","reported_seq":793,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.605918+0000","last_change":"2024-05-06T00:26:45.712444+0000","last_active":"2024-05-06T00:27:41.605918+0000","last_peered":"2024-05-06T00:27:41.605918+0000","last_clean":"2024-05-06T00:27:41.605918+0000","last_became_active":"2024-05-06T00:26:43.708328+0000","last_became_peered":"2024-05-06T00:26:43.708328+0000","last_unstale":"2024-05-06T00:27:41.605918+0000","last_undegraded":"2024-05-06T00:27:41.605918+0000","last_fullsized":"2024-05-06T00:27:41.605918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:35:35.991321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":23780864,"num_objects":21,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":11,"num_read":615,"num_read_kb":5308,"num_write":90,"num_write_kb":1111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.609293+0000","last_change":"2024-05-06T00:27:21.957697+0000","last_active":"2024-05-06T00:27:41.609293+0000","last_peered":"2024-05-06T00:27:41.609293+0000","last_clean":"2024-05-06T00:27:41.609293+0000","last_became_active":"2024-05-06T00:27:21.957293+0000","last_became_peered":"2024-05-06T00:27:21.957293+0000","last_unstale":"2024-05-06T00:27:41.609293+0000","last_undegraded":"2024-05-06T00:27:41.609293+0000","last_fullsized":"2024-05-06T00:27:41.609293+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:36:37.494499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"58'216","reported_seq":2685,"reported_epoch":61,"state":"peering","last_fresh":"2024-05-06T00:27:41.285817+0000","last_change":"2024-05-06T00:27:41.610056+0000","last_active":"2024-05-06T00:27:41.610056+0000","last_peered":"2024-05-06T00:27:41.285817+0000","last_clean":"2024-05-06T00:27:38.621371+0000","last_became_active":"2024-05-06T00:27:41.124757+0000","last_became_peered":"2024-05-06T00:27:41.124757+0000","last_unstale":"2024-05-06T00:27:41.285817+0000","last_undegraded":"2024-05-06T00:27:41.285817+0000","last_fullsized":"2024-05-06T00:27:41.113739+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":9375818,"num_objects":20,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":11,"num_read":3494,"num_read_kb":11004,"num_write":153,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":26698,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.604891+0000","last_change":"2024-05-06T00:27:40.124309+0000","last_active":"2024-05-06T00:27:41.604891+0000","last_peered":"2024-05-06T00:27:41.604891+0000","last_clean":"2024-05-06T00:27:41.604891+0000","last_became_active":"2024-05-06T00:27:40.123433+0000","last_became_peered":"2024-05-06T00:27:40.123433+0000","last_unstale":"2024-05-06T00:27:41.604891+0000","last_undegraded":"2024-05-06T00:27:41.604891+0000","last_fullsized":"2024-05-06T00:27:41.604891+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:54:51.350069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":137,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.606299+0000","last_change":"2024-05-06T00:27:21.961996+0000","last_active":"2024-05-06T00:27:41.606299+0000","last_peered":"2024-05-06T00:27:41.606299+0000","last_clean":"2024-05-06T00:27:41.606299+0000","last_became_active":"2024-05-06T00:27:21.961515+0000","last_became_peered":"2024-05-06T00:27:21.961515+0000","last_unstale":"2024-05-06T00:27:41.606299+0000","last_undegraded":"2024-05-06T00:27:41.606299+0000","last_fullsized":"2024-05-06T00:27:41.606299+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:36:39.984864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"57'293","reported_seq":1688,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.604681+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:27:41.604681+0000","last_peered":"2024-05-06T00:27:41.604681+0000","last_clean":"2024-05-06T00:27:41.604681+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:27:41.604681+0000","last_undegraded":"2024-05-06T00:27:41.604681+0000","last_fullsized":"2024-05-06T00:27:41.604681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:01:27.904013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":18699416,"num_objects":26,"num_object_clones":3,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":16,"num_read":2339,"num_read_kb":4002,"num_write":379,"num_write_kb":811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"58'129","reported_seq":3110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.603661+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:27:41.603661+0000","last_peered":"2024-05-06T00:27:41.603661+0000","last_clean":"2024-05-06T00:27:41.603661+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:27:41.603661+0000","last_undegraded":"2024-05-06T00:27:41.603661+0000","last_fullsized":"2024-05-06T00:27:41.603661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:02:22.517753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":10641998,"num_objects":31,"num_object_clones":6,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":20,"num_read":2917,"num_read_kb":7009,"num_write":134,"num_write_kb":8127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"58'310","reported_seq":4294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.603572+0000","last_change":"2024-05-06T00:27:21.968588+0000","last_active":"2024-05-06T00:27:41.603572+0000","last_peered":"2024-05-06T00:27:41.603572+0000","last_clean":"2024-05-06T00:27:41.603572+0000","last_became_active":"2024-05-06T00:27:21.967179+0000","last_became_peered":"2024-05-06T00:27:21.967179+0000","last_unstale":"2024-05-06T00:27:41.603572+0000","last_undegraded":"2024-05-06T00:27:41.603572+0000","last_fullsized":"2024-05-06T00:27:41.603572+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:35:54.754292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":20705967,"num_objects":39,"num_object_clones":8,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":19,"num_read":5429,"num_read_kb":24033,"num_write":215,"num_write_kb":1014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"57'103","reported_seq":913,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.610283+0000","last_change":"2024-05-06T00:26:45.725952+0000","last_active":"2024-05-06T00:27:41.610283+0000","last_peered":"2024-05-06T00:27:41.610283+0000","last_clean":"2024-05-06T00:27:41.610283+0000","last_became_active":"2024-05-06T00:26:43.686980+0000","last_became_peered":"2024-05-06T00:26:43.686980+0000","last_unstale":"2024-05-06T00:27:41.610283+0000","last_undegraded":"2024-05-06T00:27:41.610283+0000","last_fullsized":"2024-05-06T00:27:41.610283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:52:05.193954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":11674280,"num_objects":23,"num_object_clones":3,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":14,"num_read":762,"num_read_kb":12243,"num_write":86,"num_write_kb":4623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"61'142","reported_seq":1094,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.608989+0000","last_change":"2024-05-06T00:27:41.608989+0000","last_active":"2024-05-06T00:27:41.608989+0000","last_peered":"2024-05-06T00:27:41.365144+0000","last_clean":"2024-05-06T00:27:38.477457+0000","last_became_active":"2024-05-06T00:27:41.124748+0000","last_became_peered":"2024-05-06T00:27:41.124748+0000","last_unstale":"2024-05-06T00:27:41.608989+0000","last_undegraded":"2024-05-06T00:27:41.608989+0000","last_fullsized":"2024-05-06T00:27:41.608989+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":3849216,"num_objects":24,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":17,"num_read":792,"num_read_kb":3633,"num_write":120,"num_write_kb":876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":53248,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"58'564","reported_seq":2939,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.606397+0000","last_change":"2024-05-06T00:27:21.957565+0000","last_active":"2024-05-06T00:27:41.606397+0000","last_peered":"2024-05-06T00:27:41.606397+0000","last_clean":"2024-05-06T00:27:41.606397+0000","last_became_active":"2024-05-06T00:27:21.955822+0000","last_became_peered":"2024-05-06T00:27:21.955822+0000","last_unstale":"2024-05-06T00:27:41.606397+0000","last_undegraded":"2024-05-06T00:27:41.606397+0000","last_fullsized":"2024-05-06T00:27:41.606397+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:25:34.659013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":8757833,"num_objects":34,"num_object_clones":5,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":22,"num_read":4624,"num_read_kb":10366,"num_write":909,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"57'284","reported_seq":1852,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.604315+0000","last_change":"2024-05-06T00:27:41.604315+0000","last_active":"2024-05-06T00:27:41.604315+0000","last_peered":"2024-05-06T00:27:41.363315+0000","last_clean":"2024-05-06T00:27:35.131589+0000","last_became_active":"2024-05-06T00:27:41.134252+0000","last_became_peered":"2024-05-06T00:27:41.134252+0000","last_unstale":"2024-05-06T00:27:41.604315+0000","last_undegraded":"2024-05-06T00:27:41.604315+0000","last_fullsized":"2024-05-06T00:27:41.604315+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":11888766,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":15,"num_read":2898,"num_read_kb":7532,"num_write":391,"num_write_kb":2136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1517694,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"58'132","reported_seq":713,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.373282+0000","last_change":"2024-05-06T00:27:42.504720+0000","last_active":"2024-05-06T00:27:42.504720+0000","last_peered":"2024-05-06T00:27:41.373282+0000","last_clean":"2024-05-06T00:27:38.159025+0000","last_became_active":"2024-05-06T00:27:41.126335+0000","last_became_peered":"2024-05-06T00:27:41.126335+0000","last_unstale":"2024-05-06T00:27:41.373282+0000","last_undegraded":"2024-05-06T00:27:41.373282+0000","last_fullsized":"2024-05-06T00:27:41.110154+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":3355988,"num_objects":22,"num_object_clones":3,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":13,"num_read":738,"num_read_kb":6152,"num_write":151,"num_write_kb":540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":52052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"58'128","reported_seq":1150,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.607607+0000","last_change":"2024-05-06T00:27:41.607607+0000","last_active":"2024-05-06T00:27:41.607607+0000","last_peered":"2024-05-06T00:27:41.490556+0000","last_clean":"2024-05-06T00:27:38.222641+0000","last_became_active":"2024-05-06T00:27:41.130817+0000","last_became_peered":"2024-05-06T00:27:41.130817+0000","last_unstale":"2024-05-06T00:27:41.607607+0000","last_undegraded":"2024-05-06T00:27:41.607607+0000","last_fullsized":"2024-05-06T00:27:41.607607+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":15169024,"num_objects":22,"num_object_clones":3,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":14,"num_read":1288,"num_read_kb":4309,"num_write":99,"num_write_kb":2497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":211968,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"58'107","reported_seq":754,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.603531+0000","last_change":"2024-05-06T00:27:21.968173+0000","last_active":"2024-05-06T00:27:41.603531+0000","last_peered":"2024-05-06T00:27:41.603531+0000","last_clean":"2024-05-06T00:27:41.603531+0000","last_became_active":"2024-05-06T00:27:21.965833+0000","last_became_peered":"2024-05-06T00:27:21.965833+0000","last_unstale":"2024-05-06T00:27:41.603531+0000","last_undegraded":"2024-05-06T00:27:41.603531+0000","last_fullsized":"2024-05-06T00:27:41.603531+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:43.984786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":16378880,"num_objects":26,"num_object_clones":3,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":20,"num_read":547,"num_read_kb":5113,"num_write":115,"num_write_kb":1191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"58'158","reported_seq":2898,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-06T00:27:41.606055+0000","last_change":"2024-05-06T00:26:45.713363+0000","last_active":"2024-05-06T00:27:41.606055+0000","last_peered":"2024-05-06T00:27:41.606055+0000","last_clean":"2024-05-06T00:27:41.606055+0000","last_became_active":"2024-05-06T00:26:43.692971+0000","last_became_peered":"2024-05-06T00:26:43.692971+0000","last_unstale":"2024-05-06T00:27:41.606055+0000","last_undegraded":"2024-05-06T00:27:41.606055+0000","last_fullsized":"2024-05-06T00:27:41.606055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:32:43.160853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":21742080,"num_objects":34,"num_object_clones":8,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":19,"num_read":2668,"num_read_kb":11131,"num_write":157,"num_write_kb":1826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"58'260","reported_seq":6012,"reported_epoch":62,"state":"peering","last_fresh":"2024-05-06T00:27:41.398684+0000","last_change":"2024-05-06T00:27:42.506851+0000","last_active":"2024-05-06T00:27:42.506851+0000","last_peered":"2024-05-06T00:27:41.398684+0000","last_clean":"2024-05-06T00:27:38.102139+0000","last_became_active":"2024-05-06T00:27:41.126023+0000","last_became_peered":"2024-05-06T00:27:41.126023+0000","last_unstale":"2024-05-06T00:27:41.398684+0000","last_undegraded":"2024-05-06T00:27:41.398684+0000","last_fullsized":"2024-05-06T00:27:41.110001+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":23062113,"num_objects":27,"num_object_clones":5,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":11,"num_read":6578,"num_read_kb":22166,"num_write":191,"num_write_kb":3109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2163809,"num_keys_recovered":23,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":493086989,"num_objects":845,"num_object_clones":149,"num_object_copies":1690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":479,"num_read":80440,"num_read_kb":341901,"num_write":7348,"num_write_kb":89672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":5173094,"num_keys_recovered":138,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":50167808,"data_stored":77787485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6813,"ondisk_log_size":6813,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640276,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640274,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50708,"kb_used_data":11208,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94321132,"statfs":{"total":96636764160,"available":96584839168,"internally_reserved":0,"allocated":11476992,"data_stored":18955923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640274,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44956,"kb_used_data":9612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94326884,"statfs":{"total":96636764160,"available":96590729216,"internally_reserved":0,"allocated":9842688,"data_stored":13406167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640275,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49452,"kb_used_data":9952,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94322388,"statfs":{"total":96636764160,"available":96586125312,"internally_reserved":0,"allocated":10190848,"data_stored":13574796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640276,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51600,"kb_used_data":12100,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94320240,"statfs":{"total":96636764160,"available":96583925760,"internally_reserved":0,"allocated":12390400,"data_stored":20025876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640274,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40860,"kb_used_data":5516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330980,"statfs":{"total":96636764160,"available":96594923520,"internally_reserved":0,"allocated":5648384,"data_stored":7141185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4341760,"data_stored":6265710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11677696,"data_stored":19740769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9478144,"data_stored":13289689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8536064,"data_stored":12530692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10776576,"data_stored":18676078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5357568,"data_stored":7284547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:27:44.501 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:44.499+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:44.599 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:44.595+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:44.624 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:44.625 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:44.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:44.700 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:44.698+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:44.801 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:44.799+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:44.807 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:44.807 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:44.816 INFO:teuthology.orchestra.run.smithi186.stdout:flattening image image_client.0-clone17 2024-05-06T00:27:44.822 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:44.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:44.913 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:44.914+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:44.963 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:44.963 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:44.978 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:45.002 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:44.999+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:45.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:45.039 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:45.039 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:45.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:45.057 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:45.077 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:45.077 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:45.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:45.098 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:45.102 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:45.099+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:45.123 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:45.123 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:45.139 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:45.140 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:45.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:45.203 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:45.199+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:45.247 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-05-06T00:27:45.305 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:45.303+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:45.313 INFO:teuthology.orchestra.run.smithi186.stdout:2378 read 0x38e600 thru 0x38ed31 (0x732 bytes) 2024-05-06T00:27:45.314 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.508864889000002, 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:27:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:27:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.508848686, 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:27:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:27:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2527 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch ping cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.152754737000002, 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16552214100000001, 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2527 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160471+0000", 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.206e-06 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160477+0000", 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7440000000000001e-06 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160491+0000", 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3739e-05 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160506+0000", 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.473e-05 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160712+0000", 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020643899999999999 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160736+0000", 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3676000000000002e-05 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325517+0000", 2024-05-06T00:27:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16478073700000001 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325685+0000", 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016879300000000001 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325992+0000", 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030607699999999998 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:1976 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1561637489999992, 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050344819999999998, 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1976 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157061+0000", 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3200000000000005e-07 2024-05-06T00:27:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157064+0000", 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.022e-06 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157091+0000", 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6860999999999999e-05 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157101+0000", 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0293e-05 2024-05-06T00:27:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207258+0000", 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050156559000000003 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207333+0000", 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5426000000000006e-05 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207405+0000", 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2026999999999993e-05 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:1977 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 874496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1561124609999993, 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050463471000000003, 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1977 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5400000000000001e-07 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157116+0000", 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3720000000000001e-06 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157125+0000", 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5310000000000007e-06 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157132+0000", 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.815e-06 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207435+0000", 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050303193000000003 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207484+0000", 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8646000000000001e-05 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207575+0000", 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1260000000000004e-05 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2109 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 465920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1560832150000007, 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050607512, 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2109 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.89e-07 2024-05-06T00:27:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157144+0000", 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0340000000000001e-06 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157153+0000", 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2409999999999995e-06 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157159+0000", 2024-05-06T00:27:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6719999999999998e-06 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207604+0000", 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050444298999999998 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207674+0000", 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0636000000000006e-05 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207748+0000", 2024-05-06T00:27:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4140999999999994e-05 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2110 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1560565910000005, 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050757732999999999, 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2110 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157175+0000", 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5100000000000004e-06 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157183+0000", 2024-05-06T00:27:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3429999999999996e-06 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157189+0000", 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1140000000000002e-06 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207777+0000", 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050587886999999998 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207851+0000", 2024-05-06T00:27:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4368000000000004e-05 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207925+0000", 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4046000000000002e-05 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2111 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 486400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1560240079999993, 2024-05-06T00:27:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050893031999999998, 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2111 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157201+0000", 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7899999999999998e-07 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157204+0000", 2024-05-06T00:27:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0309999999999999e-06 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157212+0000", 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6719999999999993e-06 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157219+0000", 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1750000000000002e-06 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207953+0000", 2024-05-06T00:27:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050734855000000002 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208022+0000", 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8468999999999998e-05 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208093+0000", 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1250999999999999e-05 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2112 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 496640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1559970059999998, 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051032028, 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2112 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157228+0000", 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8999999999999997e-07 2024-05-06T00:27:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157230+0000", 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8049999999999998e-06 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157239+0000", 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2889999999999998e-06 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157245+0000", 2024-05-06T00:27:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3389999999999998e-06 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208121+0000", 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050875709999999998 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208189+0000", 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8711e-05 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208259+0000", 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9684000000000006e-05 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2113 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1559708020000006, 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051203918000000001, 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2113 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157655+0000", 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040158000000000001 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157661+0000", 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5840000000000002e-06 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157675+0000", 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4354e-05 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157684+0000", 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1859999999999998e-06 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208288+0000", 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050603883000000002 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208357+0000", 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8884e-05 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208457+0000", 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100447 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2114 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 517120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.155523401, 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050919006000000003, 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2114 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158519+0000", 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00081808800000000002 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158524+0000", 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.558e-06 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158538+0000", 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3703000000000001e-05 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158548+0000", 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0180000000000001e-05 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208486+0000", 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049937466 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208551+0000", 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-05 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208620+0000", 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8281000000000006e-05 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2116 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 537600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1546268099999999, 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050356630999999999, 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2116 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158598+0000", 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7700000000000004e-07 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158602+0000", 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4300000000000002e-06 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158610+0000", 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6510000000000007e-06 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158616+0000", 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1330000000000004e-06 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208813+0000", 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050196527999999997 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208881+0000", 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7718000000000005e-05 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208954+0000", 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3393999999999996e-05 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2117 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 547840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1545977490000006, 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050509317999999997, 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2117 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158627+0000", 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3200000000000005e-07 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158656+0000", 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8989999999999999e-05 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158666+0000", 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0093e-05 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158673+0000", 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3170000000000004e-06 2024-05-06T00:27:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208987+0000", 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050313331000000003 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209059+0000", 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2151999999999996e-05 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209136+0000", 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6803000000000003e-05 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2118 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 558080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1545388469999995, 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.05062726, 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2118 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158686+0000", 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158689+0000", 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0929999999999999e-06 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158697+0000", 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8560000000000007e-06 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158704+0000", 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8970000000000002e-06 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209167+0000", 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050462662999999998 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209239+0000", 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2700999999999994e-05 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209313+0000", 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3301999999999997e-05 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2119 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 568320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1545105000000007, 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050765355999999998, 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2119 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7899999999999998e-07 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158717+0000", 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1140000000000002e-06 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158726+0000", 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2470000000000008e-06 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158735+0000", 2024-05-06T00:27:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0429999999999996e-06 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209342+0000", 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050607632999999999 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209415+0000", 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2675999999999993e-05 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209479+0000", 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4064000000000002e-05 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2120 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 578560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1544794389999993, 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050922111999999999, 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2120 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158770+0000", 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4830000000000001e-05 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158774+0000", 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.656e-06 2024-05-06T00:27:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158784+0000", 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3729999999999999e-06 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158791+0000", 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1860000000000003e-06 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209512+0000", 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050721016000000001 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209593+0000", 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1629000000000001e-05 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209667+0000", 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3422000000000003e-05 2024-05-06T00:27:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2121 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 588800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1544224280000002, 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051033950000000002, 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2121 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162924+0000", 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041224440000000003 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162931+0000", 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7909999999999999e-06 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162943+0000", 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2009e-05 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162952+0000", 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1160000000000005e-06 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209698+0000", 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046745791000000002 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209770+0000", 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2230000000000005e-05 2024-05-06T00:27:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209836+0000", 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5568999999999995e-05 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2149 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 875520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.1493507330000003, 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050369018000000002, 2024-05-06T00:27:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2149 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:45.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163874+0000", 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9699999999999996e-07 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163877+0000", 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.497e-06 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163885+0000", 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0539999999999995e-06 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163891+0000", 2024-05-06T00:27:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2639999999999997e-06 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214165+0000", 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050274513999999999 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214202+0000", 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6390000000000002e-05 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214242+0000", 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0701999999999998e-05 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4955.0:5833 3.c 3:361ba281:::rbd_data.110ce7c6f85.0000000000000048:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.652744153, 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.95602562599999996, 2024-05-06T00:27:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4955", 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5833 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660487+0000", 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0979999999999996e-06 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660495+0000", 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2989999999999997e-06 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660514+0000", 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8190999999999999e-05 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660526+0000", 2024-05-06T00:27:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2892e-05 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660667+0000", 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014097599999999999 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660686+0000", 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.889e-05 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:42.616398+0000", 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.95571166399999996 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:42.616506+0000", 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010761599999999999 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4955.0:5834 3.c 3:361ba281:::rbd_data.110ce7c6f85.0000000000000048:head [read 4096~28672] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.652639282, 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.95599441399999996, 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4955", 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5834 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660586+0000", 2024-05-06T00:27:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660595+0000", 2024-05-06T00:27:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2969999999999992e-06 2024-05-06T00:27:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660607+0000", 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1685000000000001e-05 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660615+0000", 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6610000000000006e-06 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660776+0000", 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016069399999999999 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660795+0000", 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9398e-05 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:42.616529+0000", 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.95573370700000004 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:42.616579+0000", 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0142999999999997e-05 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.411 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:45.413 INFO:teuthology.orchestra.run.smithi186.stdout:2379 punch from 0x829200 to 0x82ce00, (0x3c00 bytes) 2024-05-06T00:27:45.413 INFO:teuthology.orchestra.run.smithi186.stdout:2381 trunc from 0x9a8000 to 0x277fe00 2024-05-06T00:27:45.413 INFO:teuthology.orchestra.run.smithi186.stdout:2385 punch from 0x1feca00 to 0x1ff1000, (0x4600 bytes) 2024-05-06T00:27:45.414 INFO:teuthology.orchestra.run.smithi186.stdout:2386 read 0x1d1000 thru 0x1dcd3c (0xbd3d bytes) 2024-05-06T00:27:45.414 INFO:teuthology.orchestra.run.smithi186.stdout:2390 read 0xa09a00 thru 0xa11b36 (0x8137 bytes) 2024-05-06T00:27:45.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:45.414 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:45.414+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:45.425 INFO:teuthology.orchestra.run.smithi186.stdout:2391 trunc from 0x277fe00 to 0x770b800 2024-05-06T00:27:45.448 INFO:teuthology.orchestra.run.smithi186.stdout:2395 write 0x50fec00 thru 0x5108d44 (0xa145 bytes) 2024-05-06T00:27:45.463 INFO:teuthology.orchestra.run.smithi186.stdout:2396 trunc from 0x770b800 to 0x1d35000 2024-05-06T00:27:45.513 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:45.514+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:45.523 INFO:teuthology.orchestra.run.smithi186.stdout:2397 punch from 0x1bd5000 to 0x1be0200, (0xb200 bytes) 2024-05-06T00:27:45.525 INFO:teuthology.orchestra.run.smithi186.stdout:2398 read 0x10e3600 thru 0x10efdc4 (0xc7c5 bytes) 2024-05-06T00:27:45.551 INFO:teuthology.orchestra.run.smithi186.stdout:2399 write 0x431a200 thru 0x43276dd (0xd4de bytes) 2024-05-06T00:27:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:45.592 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:45.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:45.614 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:45.611+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:45.644 INFO:teuthology.orchestra.run.smithi186.stdout:2401 write 0xc566200 thru 0xc56c722 (0x6523 bytes) 2024-05-06T00:27:45.645 INFO:teuthology.orchestra.run.smithi186.stdout:2403 read 0xb641800 thru 0xb6433b4 (0x1bb5 bytes) 2024-05-06T00:27:45.645 INFO:teuthology.orchestra.run.smithi186.stdout:2404 write 0x5f6de00 thru 0x5f70e50 (0x3051 bytes) 2024-05-06T00:27:45.645 INFO:teuthology.orchestra.run.smithi186.stdout:2405 read 0xb82ac00 thru 0xb8372c6 (0xc6c7 bytes) 2024-05-06T00:27:45.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:27:45.653 INFO:teuthology.orchestra.run.smithi186.stdout:2407 trunc from 0xc56c800 to 0x92a0e00 2024-05-06T00:27:45.694 INFO:teuthology.orchestra.run.smithi186.stdout:2408 write 0x6295600 thru 0x62a0a85 (0xb486 bytes) 2024-05-06T00:27:45.697 INFO:teuthology.orchestra.run.smithi186.stdout:2413 read 0x7c88200 thru 0x7c8aad4 (0x28d5 bytes) 2024-05-06T00:27:45.698 INFO:teuthology.orchestra.run.smithi186.stdout:2414 flatten 2024-05-06T00:27:45.715 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:45.711+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:45.721 INFO:teuthology.orchestra.run.smithi186.stdout:2415 write 0x45ec00 thru 0x466b62 (0x7f63 bytes) 2024-05-06T00:27:45.723 INFO:teuthology.orchestra.run.smithi186.stdout:2417 clone 19 order 20 su 4096 sc 6 2024-05-06T00:27:45.778 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:45.778 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:45.793 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:45.816 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:45.811+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:45.912 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2390 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 647168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7392265840000007, 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051610458999999997, 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2390 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171462+0000", 2024-05-06T00:27:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0100000000000004e-07 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171465+0000", 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0580000000000002e-06 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171472+0000", 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5989999999999996e-06 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171478+0000", 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8919999999999997e-06 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.222974+0000", 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051496171 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223025+0000", 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0377000000000002e-05 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223071+0000", 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6560999999999997e-05 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2391 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7390516169999994, 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051534153999999999, 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2391 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171637+0000", 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9800000000000003e-07 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171641+0000", 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2830000000000003e-06 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171651+0000", 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7219999999999994e-06 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171658+0000", 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1840000000000002e-06 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223088+0000", 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051430590999999998 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223127+0000", 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8921000000000002e-05 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223170+0000", 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2654999999999997e-05 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2399 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 739328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7380060200000003, 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051514772, 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2399 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172682+0000", 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0800000000000004e-07 2024-05-06T00:27:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172684+0000", 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3910000000000001e-06 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172691+0000", 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3999999999999997e-06 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172696+0000", 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5330000000000002e-06 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224101+0000", 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051404584000000003 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224156+0000", 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4617000000000002e-05 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224196+0000", 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0639000000000002e-05 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2400 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 749568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7379831869999993, 2024-05-06T00:27:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051590648000000003, 2024-05-06T00:27:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2400 2024-05-06T00:27:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172705+0000", 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172707+0000", 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6350000000000002e-06 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172715+0000", 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4660000000000002e-06 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172720+0000", 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5249999999999996e-06 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224216+0000", 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051495763999999999 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224253+0000", 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6529999999999998e-05 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224295+0000", 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2190000000000001e-05 2024-05-06T00:27:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2401 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 759808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7379601049999991, 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051663232000000003, 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2401 2024-05-06T00:27:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172728+0000", 2024-05-06T00:27:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8400000000000005e-07 2024-05-06T00:27:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172730+0000", 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3010000000000001e-06 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172736+0000", 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-06 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172742+0000", 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5260000000000001e-06 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224320+0000", 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051578218000000002 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224357+0000", 2024-05-06T00:27:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6418000000000003e-05 2024-05-06T00:27:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224391+0000", 2024-05-06T00:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3974999999999999e-05 2024-05-06T00:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2402 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 770048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7379387560000001, 2024-05-06T00:27:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051756008999999999, 2024-05-06T00:27:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2402 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7100000000000001e-07 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172752+0000", 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.238e-06 2024-05-06T00:27:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172758+0000", 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5989999999999998e-06 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172764+0000", 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6030000000000004e-06 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224406+0000", 2024-05-06T00:27:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051642527000000001 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224444+0000", 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7251e-05 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224505+0000", 2024-05-06T00:27:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1119999999999998e-05 2024-05-06T00:27:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2403 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7379171749999998, 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051822071999999997, 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2403 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172771+0000", 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9599999999999999e-07 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172791+0000", 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9826e-05 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172799+0000", 2024-05-06T00:27:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2109999999999998e-06 2024-05-06T00:27:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172806+0000", 2024-05-06T00:27:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7479999999999996e-06 2024-05-06T00:27:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224522+0000", 2024-05-06T00:27:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051716744000000002 2024-05-06T00:27:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224559+0000", 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6622999999999998e-05 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224592+0000", 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3423999999999997e-05 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2404 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 790528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.737868186, 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051892201999999998, 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2404 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172820+0000", 2024-05-06T00:27:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1600000000000001e-07 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172823+0000", 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7300000000000001e-06 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172831+0000", 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4120000000000001e-06 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172837+0000", 2024-05-06T00:27:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5260000000000001e-06 2024-05-06T00:27:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224608+0000", 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051771073000000001 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224664+0000", 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5948e-05 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224711+0000", 2024-05-06T00:27:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7797000000000003e-05 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2405 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 800768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7378266690000004, 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051955289000000002, 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2405 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3099999999999998e-07 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172864+0000", 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8600000000000001e-06 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172872+0000", 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1049999999999995e-06 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172879+0000", 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4359999999999999e-06 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224728+0000", 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051849004999999997 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224765+0000", 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7252000000000002e-05 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224816+0000", 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1100000000000002e-05 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2406 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 811008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7377984869999992, 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052018992, 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2406 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175329+0000", 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024402790000000001 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175335+0000", 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9229999999999999e-06 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175344+0000", 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9029999999999993e-06 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175350+0000", 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3430000000000001e-06 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224833+0000", 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049482544000000003 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224873+0000", 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9882999999999997e-05 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224908+0000", 2024-05-06T00:27:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5117000000000003e-05 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4204 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3433635640000006, 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053179637000000002, 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4204 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567325+0000", 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8299999999999995e-07 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567329+0000", 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4730000000000002e-06 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567340+0000", 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.027e-05 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567348+0000", 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0199999999999994e-06 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614100+0000", 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046752557 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614266+0000", 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000165878 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614835+0000", 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00056892300000000004 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619066+0000", 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042308729999999996 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620394+0000", 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013280080000000001 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620443+0000", 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9017000000000002e-05 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620504+0000", 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0735e-05 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4205 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3433225380000007, 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054249551999999999, 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4205 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567366+0000", 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567370+0000", 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2030000000000002e-06 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567396+0000", 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6274e-05 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567405+0000", 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7860000000000002e-06 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615526+0000", 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.048120586 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615603+0000", 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7948000000000007e-05 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615884+0000", 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028067199999999999 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620576+0000", 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046916049999999997 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621505+0000", 2024-05-06T00:27:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00092975999999999996 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621553+0000", 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7555000000000001e-05 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621614+0000", 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1426000000000004e-05 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4227 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3346892609999994, 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054371322, 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4227 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575999+0000", 2024-05-06T00:27:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.237e-06 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576007+0000", 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3660000000000003e-06 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576025+0000", 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7934000000000001e-05 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576041+0000", 2024-05-06T00:27:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.607e-05 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619115+0000", 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043074610999999999 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619242+0000", 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012649800000000001 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619799+0000", 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000557531 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627599+0000", 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077998829999999996 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630216+0000", 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026171290000000002 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630304+0000", 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7564000000000004e-05 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630370+0000", 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5499000000000004e-05 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4228 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3345652779999995, 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054411922000000001, 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4228 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576124+0000", 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.33e-06 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576169+0000", 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5867e-05 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576194+0000", 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4536999999999999e-05 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576215+0000", 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0679999999999999e-05 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620628+0000", 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.044413312000000003 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620704+0000", 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6255999999999996e-05 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621011+0000", 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030724300000000001 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629112+0000", 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0081005490000000003 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630444+0000", 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00133248 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630492+0000", 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7144999999999997e-05 2024-05-06T00:27:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630534+0000", 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2522999999999999e-05 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4229 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3344484140000006, 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054513172999999998, 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4229 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576240+0000", 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4100000000000001e-06 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576272+0000", 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-05 2024-05-06T00:27:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576294+0000", 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2283e-05 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576311+0000", 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6651e-05 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621671+0000", 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.045360079999999997 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621747+0000", 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5434999999999999e-05 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622136+0000", 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038980800000000001 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629125+0000", 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069886499999999999 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630609+0000", 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001483563 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630691+0000", 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2144999999999994e-05 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630752+0000", 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1518000000000003e-05 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4230 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3343539, 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054593549999999998, 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4230 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576335+0000", 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5799999999999999e-06 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576343+0000", 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5599999999999996e-06 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576362+0000", 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8810000000000001e-05 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576376+0000", 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4887e-05 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622737+0000", 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046360459 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622836+0000", 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8911000000000002e-05 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.623379+0000", 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054296499999999996 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629139+0000", 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057601359999999999 2024-05-06T00:27:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630805+0000", 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016661720000000001 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630878+0000", 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3348000000000003e-05 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630927+0000", 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8721999999999999e-05 2024-05-06T00:27:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4231 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3342731830000005, 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054783287999999999, 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4231 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576416+0000", 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.28e-06 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576423+0000", 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576443+0000", 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.984e-05 2024-05-06T00:27:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576459+0000", 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6223e-05 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624037+0000", 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.047578143000000003 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624129+0000", 2024-05-06T00:27:46.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1753000000000001e-05 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624431+0000", 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030202000000000001 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629149+0000", 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047174089999999997 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631083+0000", 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001934758 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631148+0000", 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4845999999999995e-05 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631198+0000", 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9443000000000001e-05 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4232 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3342083000000002, 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054906561, 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4232 2024-05-06T00:27:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576481+0000", 2024-05-06T00:27:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5269999999999999e-06 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576488+0000", 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8129999999999996e-06 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576507+0000", 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8417e-05 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576523+0000", 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6546e-05 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.625281+0000", 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.04875753 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.625377+0000", 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6224000000000006e-05 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.626749+0000", 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001371585 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630390+0000", 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036414749999999999 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631265+0000", 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087439099999999997 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631333+0000", 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8534000000000005e-05 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631386+0000", 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2519000000000003e-05 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4233 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3341439400000006, 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056371963999999997, 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4233 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576545+0000", 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0669999999999999e-06 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576558+0000", 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3305999999999999e-05 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576576+0000", 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8587e-05 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576591+0000", 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4533e-05 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627664+0000", 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051072838000000002 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627956+0000", 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029239499999999998 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.628429+0000", 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047251999999999998 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632630+0000", 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042016800000000002 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632724+0000", 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3568999999999998e-05 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632839+0000", 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011482200000000001 2024-05-06T00:27:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632915+0000", 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6647e-05 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4234 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3340789229999999, 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057621897999999998, 2024-05-06T00:27:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4234 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576610+0000", 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.15e-06 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576617+0000", 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8499999999999996e-06 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576636+0000", 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9205999999999999e-05 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576650+0000", 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.416e-05 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629208+0000", 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052558219000000003 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629293+0000", 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4389999999999997e-05 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629620+0000", 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032745499999999999 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.633738+0000", 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004117725 2024-05-06T00:27:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634045+0000", 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030709200000000002 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634146+0000", 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101362 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634230+0000", 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4289000000000006e-05 2024-05-06T00:27:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.028 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:46.030 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:46.030 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:46.030+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:46.131 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:46.127+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:46.196 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:46.197 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:46.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:46.257 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:46.258+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:46.333 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:46.333 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:46.333 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:46.333 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:46.333 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:46.334 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:46.331+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:46.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:46.437 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:46.438+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:860 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3407872~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.505309873000002, 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054999124000000003, 2024-05-06T00:27:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 860 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005693+0000", 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3969999999999999e-06 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005699+0000", 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1399999999999997e-06 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005718+0000", 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9190000000000001e-05 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005733+0000", 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4303e-05 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060250+0000", 2024-05-06T00:27:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054516805000000002 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060353+0000", 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103509 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060691+0000", 2024-05-06T00:27:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033777999999999998 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:861 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.505249192000001, 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055437840000000002, 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 861 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005754+0000", 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005773+0000", 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9817e-05 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005792+0000", 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8770000000000002e-05 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005824+0000", 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1643000000000002e-05 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060742+0000", 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054918417999999997 2024-05-06T00:27:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060845+0000", 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102507 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061190+0000", 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034541500000000001 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:862 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4063232~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.505155715000001, 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055843853999999998, 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 862 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005847+0000", 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3200000000000001e-06 2024-05-06T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005854+0000", 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8299999999999998e-06 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005872+0000", 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.84e-05 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005886+0000", 2024-05-06T00:27:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.347e-05 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061241+0000", 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055355559999999998 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061343+0000", 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101335 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061690+0000", 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034693899999999999 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:863 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.505095617999999, 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056056918999999997, 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 863 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005907+0000", 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.234e-06 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005913+0000", 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8000000000000004e-06 2024-05-06T00:27:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005932+0000", 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.895e-05 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005964+0000", 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2016e-05 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061741+0000", 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055776824000000003 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061840+0000", 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9469000000000006e-05 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061963+0000", 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000122626 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:899 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2392064~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.505017498000001, 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056211016000000003, 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 899 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005985+0000", 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005992+0000", 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006011+0000", 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8669999999999999e-05 2024-05-06T00:27:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006024+0000", 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-05 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062007+0000", 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055982453000000001 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062096+0000", 2024-05-06T00:27:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9549000000000006e-05 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062195+0000", 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8999999999999994e-05 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:900 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2850816~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504958067, 2024-05-06T00:27:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056426437000000003, 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 900 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006045+0000", 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.22e-06 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006064+0000", 2024-05-06T00:27:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8997000000000001e-05 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006083+0000", 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9449999999999998e-05 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006098+0000", 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.451e-05 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062245+0000", 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056147619000000003 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062346+0000", 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100541 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062470+0000", 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012410000000000001 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:901 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504883986999999, 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056582556999999999, 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 901 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006119+0000", 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006125+0000", 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.207e-06 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006143+0000", 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.821e-05 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006157+0000", 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4197e-05 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062513+0000", 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056355322999999999 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062600+0000", 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7060999999999999e-05 2024-05-06T00:27:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062700+0000", 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100369 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:902 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504754546000001, 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056710658999999997, 2024-05-06T00:27:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 902 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006249+0000", 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5939999999999999e-06 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006256+0000", 2024-05-06T00:27:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1029999999999996e-06 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006277+0000", 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1637e-05 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006293+0000", 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.525e-05 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062749+0000", 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056456081999999998 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062845+0000", 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6194999999999996e-05 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062958+0000", 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000112798 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:903 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3702784~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504686821, 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056894469000000003, 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 903 2024-05-06T00:27:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006316+0000", 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2759999999999999e-06 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006322+0000", 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9839999999999998e-06 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006340+0000", 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8465999999999999e-05 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006354+0000", 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3137e-05 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063007+0000", 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056652965 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063100+0000", 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3638e-05 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063209+0000", 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109003 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:904 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4161536~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504627778, 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057092006000000001, 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 904 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006375+0000", 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006381+0000", 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0700000000000003e-06 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006399+0000", 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8372999999999998e-05 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006413+0000", 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3917000000000001e-05 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063258+0000", 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056844680000000002 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063352+0000", 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3991000000000002e-05 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063466+0000", 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113768 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:905 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4358144~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504570114, 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057200123999999998, 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 905 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006433+0000", 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006438+0000", 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5770000000000001e-06 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006456+0000", 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006469+0000", 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2666999999999999e-05 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063509+0000", 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057039566999999999 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063563+0000", 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4360000000000001e-05 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063632+0000", 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8663000000000003e-05 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:921 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.50451486, 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057645638999999999, 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 921 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006488+0000", 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0699999999999999e-06 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006493+0000", 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2159999999999997e-06 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006511+0000", 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8454e-05 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006525+0000", 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.342e-05 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063673+0000", 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057148440000000002 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063765+0000", 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1608999999999997e-05 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064132+0000", 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036743000000000002 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:922 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2785280~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504459316999998, 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058081037000000002, 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 922 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006543+0000", 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.127e-06 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006568+0000", 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4777e-05 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006588+0000", 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0203999999999998e-05 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006616+0000", 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7625999999999999e-05 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064169+0000", 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057553460000000001 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064264+0000", 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4773999999999998e-05 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064623+0000", 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000359069 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:923 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3244032~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504364183, 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058440913999999997, 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 923 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006639+0000", 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006657+0000", 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8692999999999999e-05 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006677+0000", 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9760999999999999e-05 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006692+0000", 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4467000000000001e-05 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064658+0000", 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057966692 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064746+0000", 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7851999999999998e-05 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065078+0000", 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033224200000000001 2024-05-06T00:27:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504290482000002, 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:27:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:27:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.504061910000001, 2024-05-06T00:27:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:27:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.503984626000001, 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:27:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.503788645, 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2006 3.d 3:b03dcdad:::rbd_data.110ce7c6f85.0000000000000002:13 [read 718848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3573651869999992, 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054771050000000002, 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2006 2024-05-06T00:27:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153637+0000", 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1999999999999999e-07 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153640+0000", 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.695e-06 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153647+0000", 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9619999999999996e-06 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153658+0000", 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1249e-05 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208250+0000", 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054591955999999997 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208323+0000", 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208407+0000", 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4320999999999994e-05 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2007 3.d 3:b03dcdad:::rbd_data.110ce7c6f85.0000000000000002:13 [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3573355790000008, 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054911852999999997, 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2007 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155145+0000", 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001478909 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155151+0000", 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2840000000000003e-06 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155166+0000", 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4314000000000001e-05 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155174+0000", 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3629999999999994e-06 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208436+0000", 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053262359000000002 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208505+0000", 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8677999999999995e-05 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208578+0000", 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2946000000000003e-05 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:46.600 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:46.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:46.603 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:46.599+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:46.648 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:27:46.702 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:46.699+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:46.738 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:46.739 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:46.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:46.804 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:46.802+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:46.905 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:46.903+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:46.920 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:46.920 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:46.920 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:46.920 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:46.920 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:46.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:47.006 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:47.006+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:47.107 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:47.107 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.50218787, 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.229539170999999, 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:27:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.229503995000002, 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.121 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:27:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.229415316000001, 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:27:47.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:27:47.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:27:47.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.229313373, 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:27:47.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.139 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.229204271, 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:47.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:47.145 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.229106412, 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:47.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:27:47.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:47.151 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.229041362, 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:27:47.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:27:47.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.228948716000001, 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:27:47.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:27:47.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.165 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:47.165 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:47.165 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.228876712000002, 2024-05-06T00:27:47.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:27:47.165 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:27:47.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.469415225999999, 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:27:47.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:27:47.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.469395978000001, 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.177 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:47.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:27:47.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:27:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.469345858000001, 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:27:47.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.188 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.469324807, 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.189 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:27:47.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:27:47.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:27:47.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:27:47.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.195 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.469309488, 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:27:47.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:27:47.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.469292606, 2024-05-06T00:27:47.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:47.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:27:47.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:27:47.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.46927427, 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.207 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:27:47.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:27:47.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:27:47.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.469258268000001, 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.213 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:27:47.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:27:47.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.469171905, 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.219 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:27:47.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:27:47.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:27:47.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.459209429000001, 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:27:47.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:27:47.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:47.230 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:47.231 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:47.103+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:47.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:47.232 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:47.273 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:47.271+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:47.289 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-05-06T00:27:47.334 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:47.331+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:47.394 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:47.394 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:47.405 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:47.406 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:47.411 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:47.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:47.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:47.449 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:47.449 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:47.461 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:47.459+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:47.466 INFO:teuthology.orchestra.run.smithi186.stdout:2418 trunc from 0x92a0e00 to 0xe0e6000 2024-05-06T00:27:47.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:47.487 INFO:teuthology.orchestra.run.smithi186.stdout:2419 read 0x842ba00 thru 0x8434c34 (0x9235 bytes) 2024-05-06T00:27:47.489 INFO:teuthology.orchestra.run.smithi186.stdout:2422 read 0x5245e00 thru 0x52534ad (0xd6ae bytes) 2024-05-06T00:27:47.490 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:47.491 INFO:teuthology.orchestra.run.smithi186.stdout:2423 write 0x56bc600 thru 0x56bc6c6 (0xc7 bytes) 2024-05-06T00:27:47.491 INFO:teuthology.orchestra.run.smithi186.stdout:2424 read 0x69e6200 thru 0x69f4d45 (0xeb46 bytes) 2024-05-06T00:27:47.491 INFO:teuthology.orchestra.run.smithi186.stdout:2425 punch from 0x36c3800 to 0x36cc600, (0x8e00 bytes) 2024-05-06T00:27:47.504 INFO:teuthology.orchestra.run.smithi186.stdout:2426 punch from 0x294ea00 to 0x2951e00, (0x3400 bytes) 2024-05-06T00:27:47.517 INFO:teuthology.orchestra.run.smithi186.stdout:2427 read 0x9093a00 thru 0x90a0885 (0xce86 bytes) 2024-05-06T00:27:47.517 INFO:teuthology.orchestra.run.smithi186.stdout:2428 write 0x30aa800 thru 0x30b1f6e (0x776f bytes) 2024-05-06T00:27:47.517 INFO:teuthology.orchestra.run.smithi186.stdout:2430 read 0x30e1c00 thru 0x30eb4d2 (0x98d3 bytes) 2024-05-06T00:27:47.517 INFO:teuthology.orchestra.run.smithi186.stdout:2431 read 0x6389000 thru 0x639756f (0xe570 bytes) 2024-05-06T00:27:47.530 INFO:teuthology.orchestra.run.smithi186.stdout:2432 trunc from 0xe0e6000 to 0x91cd600 2024-05-06T00:27:47.536 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:47.531+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:47.594 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:47.595 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:47.607 INFO:teuthology.orchestra.run.smithi186.stdout:2437 read 0x5482400 thru 0x5483b03 (0x1704 bytes) 2024-05-06T00:27:47.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:47.624 INFO:teuthology.orchestra.run.smithi186.stdout:2440 write 0x9a9b600 thru 0x9a9c0c8 (0xac9 bytes) 2024-05-06T00:27:47.627 INFO:teuthology.orchestra.run.smithi186.stdout:2441 punch from 0x7160000 to 0x7163c00, (0x3c00 bytes) 2024-05-06T00:27:47.641 INFO:teuthology.orchestra.run.smithi186.stdout:2442 write 0x3a4fa00 thru 0x3a54385 (0x4986 bytes) 2024-05-06T00:27:47.642 INFO:teuthology.orchestra.run.smithi186.stdout:2443 punch from 0x4010600 to 0x4011400, (0xe00 bytes) 2024-05-06T00:27:47.645 INFO:teuthology.orchestra.run.smithi186.stdout:2444 write 0x2c17800 thru 0x2c19c8a (0x248b bytes) 2024-05-06T00:27:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:2447 read 0x8a22800 thru 0x8a23a3c (0x123d bytes) 2024-05-06T00:27:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:2449 punch from 0x6d53600 to 0x6d5de00, (0xa800 bytes) 2024-05-06T00:27:47.653 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:47.651+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:47.738 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:47.735+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:47.749 INFO:teuthology.orchestra.run.smithi186.stdout:2455 trunc from 0x9a9c200 to 0x214a00 2024-05-06T00:27:47.783 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:47.783 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:47.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:47.841 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:47.839+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:47.926 INFO:teuthology.orchestra.run.smithi186.stdout:2458 read 0x110200 thru 0x115e64 (0x5c65 bytes) 2024-05-06T00:27:47.931 INFO:teuthology.orchestra.run.smithi186.stdout:2459 read 0x112800 thru 0x11d36c (0xab6d bytes) 2024-05-06T00:27:47.933 INFO:teuthology.orchestra.run.smithi186.stdout:2462 punch from 0xa1c00 to 0xa5400, (0x3800 bytes) 2024-05-06T00:27:47.941 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:47.939+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:47.960 INFO:teuthology.orchestra.run.smithi186.stdout:2463 write 0x5523c00 thru 0x55272b4 (0x36b5 bytes) 2024-05-06T00:27:47.965 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:47.965 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:47.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:48.001 INFO:teuthology.orchestra.run.smithi186.stdout:2466 write 0xc95dc00 thru 0xc963a2f (0x5e30 bytes) 2024-05-06T00:27:48.003 INFO:teuthology.orchestra.run.smithi186.stdout:2467 write 0x8809400 thru 0x8815fe5 (0xcbe6 bytes) 2024-05-06T00:27:48.003 INFO:teuthology.orchestra.run.smithi186.stdout:2468 read 0x41fbc00 thru 0x420af58 (0xf359 bytes) 2024-05-06T00:27:48.004 INFO:teuthology.orchestra.run.smithi186.stdout:2470 write 0x353de00 thru 0x354cd53 (0xef54 bytes) 2024-05-06T00:27:48.004 INFO:teuthology.orchestra.run.smithi186.stdout:2472 read 0x15c4e00 thru 0x15cada8 (0x5fa9 bytes) 2024-05-06T00:27:48.005 INFO:teuthology.orchestra.run.smithi186.stdout:2473 write 0x7820e00 thru 0x782779b (0x699c bytes) 2024-05-06T00:27:48.005 INFO:teuthology.orchestra.run.smithi186.stdout:2475 punch from 0x3e85800 to 0x3e8f400, (0x9c00 bytes) 2024-05-06T00:27:48.006 INFO:teuthology.orchestra.run.smithi186.stdout:2476 punch from 0x412fa00 to 0x4131a00, (0x2000 bytes) 2024-05-06T00:27:48.018 INFO:teuthology.orchestra.run.smithi186.stdout:2479 trunc from 0xc963c00 to 0x5a67800 2024-05-06T00:27:48.069 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:48.067+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:48.103 INFO:teuthology.orchestra.run.smithi186.stdout:2483 write 0x757ac00 thru 0x758a7d4 (0xfbd5 bytes) 2024-05-06T00:27:48.103 INFO:teuthology.orchestra.run.smithi186.stdout:2489 write 0x92fc00 thru 0x938aa1 (0x8ea2 bytes) 2024-05-06T00:27:48.103 INFO:teuthology.orchestra.run.smithi186.stdout:2492 read 0x2a60000 thru 0x2a61023 (0x1024 bytes) 2024-05-06T00:27:48.104 INFO:teuthology.orchestra.run.smithi186.stdout:2493 read 0x695e200 thru 0x6966ecf (0x8cd0 bytes) 2024-05-06T00:27:48.104 INFO:teuthology.orchestra.run.smithi186.stdout:2494 punch from 0x5aa5400 to 0x5ab1a00, (0xc600 bytes) 2024-05-06T00:27:48.105 INFO:teuthology.orchestra.run.smithi186.stdout:2495 read 0x7250e00 thru 0x725da7c (0xcc7d bytes) 2024-05-06T00:27:48.105 INFO:teuthology.orchestra.run.smithi186.stdout:2496 read 0x3313400 thru 0x33229ae (0xf5af bytes) 2024-05-06T00:27:48.107 INFO:teuthology.orchestra.run.smithi186.stdout:2497 read 0x20a00 thru 0x27c35 (0x7236 bytes) 2024-05-06T00:27:48.125 INFO:teuthology.orchestra.run.smithi186.stdout:2500 read 0x29ab200 thru 0x29b2862 (0x7663 bytes) 2024-05-06T00:27:48.143 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:48.139+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:48.145 INFO:teuthology.orchestra.run.smithi186.stdout:2501 write 0x9981000 thru 0x9981d0f (0xd10 bytes) 2024-05-06T00:27:48.148 INFO:teuthology.orchestra.run.smithi186.stdout:2503 punch from 0x3c11e00 to 0x3c18200, (0x6400 bytes) 2024-05-06T00:27:48.149 INFO:teuthology.orchestra.run.smithi186.stdout:2506 trunc from 0x9981e00 to 0x9176a00 2024-05-06T00:27:48.160 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:48.160 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:48.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:48.177 INFO:teuthology.orchestra.run.smithi186.stdout:2508 punch from 0x8266000 to 0x8274000, (0xe000 bytes) 2024-05-06T00:27:48.185 INFO:teuthology.orchestra.run.smithi186.stdout:2511 trunc from 0x9176a00 to 0x5f0ce00 2024-05-06T00:27:48.222 INFO:teuthology.orchestra.run.smithi186.stdout:2514 read 0x29c3800 thru 0x29cc436 (0x8c37 bytes) 2024-05-06T00:27:48.224 INFO:teuthology.orchestra.run.smithi186.stdout:2515 write 0x2deae00 thru 0x2df5bb2 (0xadb3 bytes) 2024-05-06T00:27:48.224 INFO:teuthology.orchestra.run.smithi186.stdout:2516 read 0x3cff400 thru 0x3d0aa4b (0xb64c bytes) 2024-05-06T00:27:48.225 INFO:teuthology.orchestra.run.smithi186.stdout:2517 punch from 0x2548000 to 0x2550800, (0x8800 bytes) 2024-05-06T00:27:48.226 INFO:teuthology.orchestra.run.smithi186.stdout:2519 write 0x34dc600 thru 0x34deaa3 (0x24a4 bytes) 2024-05-06T00:27:48.227 INFO:teuthology.orchestra.run.smithi186.stdout:2520 punch from 0x132600 to 0x13ee00, (0xc800 bytes) 2024-05-06T00:27:48.227 INFO:teuthology.orchestra.run.smithi186.stdout:2522 write 0x40d0400 thru 0x40dc3d8 (0xbfd9 bytes) 2024-05-06T00:27:48.239 INFO:teuthology.orchestra.run.smithi186.stdout:2526 trunc from 0x5f0ce00 to 0xb83ae00 2024-05-06T00:27:48.244 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:48.242+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:48.255 INFO:teuthology.orchestra.run.smithi186.stdout:2528 read 0x273d000 thru 0x27413ae (0x43af bytes) 2024-05-06T00:27:48.257 INFO:teuthology.orchestra.run.smithi186.stdout:2530 read 0x4f8cc00 thru 0x4f8e523 (0x1924 bytes) 2024-05-06T00:27:48.266 INFO:teuthology.orchestra.run.smithi186.stdout:2532 trunc from 0xb83ae00 to 0x6213600 2024-05-06T00:27:48.282 INFO:teuthology.orchestra.run.smithi186.stdout:2533 punch from 0x15e3800 to 0x15f3200, (0xfa00 bytes) 2024-05-06T00:27:48.284 INFO:teuthology.orchestra.run.smithi186.stdout:2535 write 0x2b08400 thru 0x2b10556 (0x8157 bytes) 2024-05-06T00:27:48.285 INFO:teuthology.orchestra.run.smithi186.stdout:2537 punch from 0x4353a00 to 0x435d200, (0x9800 bytes) 2024-05-06T00:27:48.287 INFO:teuthology.orchestra.run.smithi186.stdout:2540 punch from 0x10cc600 to 0x10dc000, (0xfa00 bytes) 2024-05-06T00:27:48.324 INFO:teuthology.orchestra.run.smithi186.stdout:2541 write 0xaa58600 thru 0xaa5df3d (0x593e bytes) 2024-05-06T00:27:48.338 INFO:teuthology.orchestra.run.smithi186.stdout:2542 trunc from 0xaa5e000 to 0x5de2400 2024-05-06T00:27:48.345 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:48.345 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:48.345 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:48.343+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:48.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:48.377 INFO:teuthology.orchestra.run.smithi186.stdout:2543 clone 20 order 20 su 131072 sc 3 2024-05-06T00:27:48.446 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:48.443+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:48.486 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:48.486 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:48.500 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:48.548 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:48.546+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:48.619 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:48.619 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:48.632 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:48.649 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:48.647+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:48.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:27:48.750 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:48.747+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:48.784 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone19 from 0x5de2400 (overlap 0x214a00) to 0x1aa000 2024-05-06T00:27:48.842 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:48.842 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:48.851 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:48.847+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:48.865 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:48.952 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:48.947+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:48.984 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:48.984 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:48.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:49.052 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:49.047+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:49.118 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:49.119 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:49.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:49.153 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:49.151+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:49.255 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:49.254+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:49.261 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:49.261 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:49.275 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:49.292 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-05-06T00:27:49.356 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:49.351+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:49.451 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:49.452 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:49.456 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:49.451+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:49.460 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:49.460 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:27:49.461 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4] 2024-05-06T00:27:49.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-05-06T00:27:49.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:49.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:49.557 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:49.558+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:49.644 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:49.644 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:27:49.658 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:49.658+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:49.670 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:49.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:49.704 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:49.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:49.722 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:49.740 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:49.741 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:49.759 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:49.758+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:49.765 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:49.787 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:49.787 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:49.799 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-06T00:27:49.798+0000 7fdfb36a9640 -1 mon.a@0(leader).osd e68 definitely_dead 0 2024-05-06T00:27:49.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:49.811 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:49.831 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:49.832 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:860 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3407872~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.822620326999999, 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054999124000000003, 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 860 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005693+0000", 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3969999999999999e-06 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005699+0000", 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1399999999999997e-06 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005718+0000", 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9190000000000001e-05 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005733+0000", 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4303e-05 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060250+0000", 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054516805000000002 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060353+0000", 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103509 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060691+0000", 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033777999999999998 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:861 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.837 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.822559645999998, 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055437840000000002, 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 861 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005754+0000", 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005773+0000", 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9817e-05 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005792+0000", 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8770000000000002e-05 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005824+0000", 2024-05-06T00:27:49.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1643000000000002e-05 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060742+0000", 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054918417999999997 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060845+0000", 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102507 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061190+0000", 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034541500000000001 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:862 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4063232~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.822466168999998, 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055843853999999998, 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 862 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:49.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005847+0000", 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3200000000000001e-06 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005854+0000", 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8299999999999998e-06 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005872+0000", 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.84e-05 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005886+0000", 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.347e-05 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061241+0000", 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055355559999999998 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061343+0000", 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101335 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061690+0000", 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034693899999999999 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:863 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:49.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.822406072, 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056056918999999997, 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 863 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005907+0000", 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.234e-06 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005913+0000", 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8000000000000004e-06 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005932+0000", 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.895e-05 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005964+0000", 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2016e-05 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061741+0000", 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055776824000000003 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061840+0000", 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9469000000000006e-05 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061963+0000", 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000122626 2024-05-06T00:27:49.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:899 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2392064~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.822327951999998, 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056211016000000003, 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 899 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005985+0000", 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005992+0000", 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006011+0000", 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8669999999999999e-05 2024-05-06T00:27:49.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006024+0000", 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-05 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062007+0000", 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055982453000000001 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062096+0000", 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9549000000000006e-05 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062195+0000", 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8999999999999994e-05 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:900 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2850816~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.822268521000002, 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056426437000000003, 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 900 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006045+0000", 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.22e-06 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006064+0000", 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8997000000000001e-05 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006083+0000", 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9449999999999998e-05 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006098+0000", 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.451e-05 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062245+0000", 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056147619000000003 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062346+0000", 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100541 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062470+0000", 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012410000000000001 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:901 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.822194441000001, 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056582556999999999, 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 901 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006119+0000", 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006125+0000", 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.207e-06 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006143+0000", 2024-05-06T00:27:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.821e-05 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006157+0000", 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4197e-05 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062513+0000", 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056355322999999999 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062600+0000", 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7060999999999999e-05 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062700+0000", 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100369 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:902 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.822064999999998, 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056710658999999997, 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 902 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006249+0000", 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5939999999999999e-06 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006256+0000", 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1029999999999996e-06 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006277+0000", 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1637e-05 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006293+0000", 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.525e-05 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062749+0000", 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056456081999999998 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062845+0000", 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6194999999999996e-05 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062958+0000", 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000112798 2024-05-06T00:27:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:903 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3702784~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821997275000001, 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056894469000000003, 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 903 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006316+0000", 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2759999999999999e-06 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006322+0000", 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9839999999999998e-06 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006340+0000", 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8465999999999999e-05 2024-05-06T00:27:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006354+0000", 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3137e-05 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063007+0000", 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056652965 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063100+0000", 2024-05-06T00:27:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3638e-05 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063209+0000", 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109003 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:904 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4161536~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821938232000001, 2024-05-06T00:27:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057092006000000001, 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 904 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006375+0000", 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006381+0000", 2024-05-06T00:27:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0700000000000003e-06 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006399+0000", 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8372999999999998e-05 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006413+0000", 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3917000000000001e-05 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063258+0000", 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056844680000000002 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063352+0000", 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3991000000000002e-05 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063466+0000", 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113768 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:905 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4358144~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821880568000001, 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057200123999999998, 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 905 2024-05-06T00:27:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006433+0000", 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006438+0000", 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5770000000000001e-06 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006456+0000", 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006469+0000", 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2666999999999999e-05 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063509+0000", 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057039566999999999 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063563+0000", 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4360000000000001e-05 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063632+0000", 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8663000000000003e-05 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:921 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821825314000002, 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057645638999999999, 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 921 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006488+0000", 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0699999999999999e-06 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006493+0000", 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2159999999999997e-06 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006511+0000", 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8454e-05 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006525+0000", 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.342e-05 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063673+0000", 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057148440000000002 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063765+0000", 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1608999999999997e-05 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064132+0000", 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036743000000000002 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:922 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2785280~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821769771, 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058081037000000002, 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 922 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006543+0000", 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.127e-06 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006568+0000", 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4777e-05 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006588+0000", 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0203999999999998e-05 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006616+0000", 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7625999999999999e-05 2024-05-06T00:27:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064169+0000", 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057553460000000001 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064264+0000", 2024-05-06T00:27:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4773999999999998e-05 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064623+0000", 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000359069 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:923 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3244032~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821674637000001, 2024-05-06T00:27:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058440913999999997, 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 923 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006639+0000", 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006657+0000", 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8692999999999999e-05 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006677+0000", 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9760999999999999e-05 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006692+0000", 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4467000000000001e-05 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064658+0000", 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057966692 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064746+0000", 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7851999999999998e-05 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065078+0000", 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033224200000000001 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821600935999999, 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821372363999998, 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821295079999999, 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:27:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.821099099000001, 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:27:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2006 3.d 3:b03dcdad:::rbd_data.110ce7c6f85.0000000000000002:13 [read 718848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.674675641, 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054771050000000002, 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2006 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153637+0000", 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1999999999999999e-07 2024-05-06T00:27:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153640+0000", 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.695e-06 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153647+0000", 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9619999999999996e-06 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153658+0000", 2024-05-06T00:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1249e-05 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208250+0000", 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054591955999999997 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208323+0000", 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208407+0000", 2024-05-06T00:27:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4320999999999994e-05 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2007 3.d 3:b03dcdad:::rbd_data.110ce7c6f85.0000000000000002:13 [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.674646033, 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054911852999999997, 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2007 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155145+0000", 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001478909 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155151+0000", 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2840000000000003e-06 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155166+0000", 2024-05-06T00:27:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4314000000000001e-05 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155174+0000", 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3629999999999994e-06 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208436+0000", 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053262359000000002 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208505+0000", 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8677999999999995e-05 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208578+0000", 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2946000000000003e-05 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:49.922 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:49.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:49.965 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:49.966+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:50.027 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:50.026+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:50.054 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:50.054 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:50.054 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:50.054 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:50.068 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:50.128 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:50.126+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:50.190 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:50.190 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:50.191 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:50.191 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:50.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:50.224 INFO:teuthology.orchestra.run.smithi186.stdout:2546 punch from 0x697200 to 0x69b000, (0x3e00 bytes) 2024-05-06T00:27:50.247 INFO:teuthology.orchestra.run.smithi186.stdout:2547 write 0x34db000 thru 0x34e299c (0x799d bytes) 2024-05-06T00:27:50.248 INFO:teuthology.orchestra.run.smithi186.stdout:2550 punch from 0x2649600 to 0x2659000, (0xfa00 bytes) 2024-05-06T00:27:50.250 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:50.247+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:50.330 INFO:teuthology.orchestra.run.smithi186.stdout:2551 trunc from 0x5de2400 to 0xe4c0600 2024-05-06T00:27:50.331 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:50.327+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:50.332 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:50.332 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:50.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:50.332 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:50.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.332 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.727025562999998, 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:27:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:27:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:27:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:27:50.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.454376863999997, 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:27:50.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:27:50.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.454341688, 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:27:50.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:27:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:27:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.454253008999999, 2024-05-06T00:27:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:27:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:27:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.454151066000001, 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:27:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:27:50.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:27:50.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:27:50.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:27:50.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:27:50.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.454041963999998, 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.365 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:27:50.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:27:50.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:27:50.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.371 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.453944104999998, 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:27:50.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:27:50.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.453879055000002, 2024-05-06T00:27:50.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:27:50.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:27:50.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.453786409000003, 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.383 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:27:50.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:27:50.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.453714404999999, 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.389 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:27:50.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:27:50.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:27:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:27:50.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694252919, 2024-05-06T00:27:50.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:27:50.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:27:50.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:27:50.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:27:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694233670999999, 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.400 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:27:50.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:27:50.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:27:50.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694183550999998, 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:27:50.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:27:50.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:27:50.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694162500000001, 2024-05-06T00:27:50.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:27:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694147180999998, 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:27:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:27:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694130299000001, 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:27:50.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:27:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694111963000001, 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:27:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:27:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694095960999999, 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:27:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.694009598000001, 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:27:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.684047121999999, 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:27:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.452 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:50.453 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:50.453 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:50.455 INFO:teuthology.orchestra.run.smithi186.stdout:2554 read 0xc57d400 thru 0xc58c8e4 (0xf4e5 bytes) 2024-05-06T00:27:50.455 INFO:teuthology.orchestra.run.smithi186.stdout:2556 read 0x7faf200 thru 0x7fbcea8 (0xdca9 bytes) 2024-05-06T00:27:50.455 INFO:teuthology.orchestra.run.smithi186.stdout:2561 trunc from 0xe4c0600 to 0x569a00 2024-05-06T00:27:50.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:50.456 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:50.451+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:50.494 INFO:teuthology.orchestra.run.smithi186.stdout:2564 write 0x1825a00 thru 0x1826214 (0x815 bytes) 2024-05-06T00:27:50.496 INFO:teuthology.orchestra.run.smithi186.stdout:2567 read 0xe1b600 thru 0xe215aa (0x5fab bytes) 2024-05-06T00:27:50.547 INFO:teuthology.orchestra.run.smithi186.stdout:2568 write 0xd8df000 thru 0xd8ece12 (0xde13 bytes) 2024-05-06T00:27:50.548 INFO:teuthology.orchestra.run.smithi186.stdout:2569 write 0xd83000 thru 0xd9037c (0xd37d bytes) 2024-05-06T00:27:50.548 INFO:teuthology.orchestra.run.smithi186.stdout:2572 read 0x94c3a00 thru 0x94d20b8 (0xe6b9 bytes) 2024-05-06T00:27:50.548 INFO:teuthology.orchestra.run.smithi186.stdout:2573 write 0xb216600 thru 0xb224a1e (0xe41f bytes) 2024-05-06T00:27:50.548 INFO:teuthology.orchestra.run.smithi186.stdout:2574 punch from 0x142b200 to 0x1439400, (0xe200 bytes) 2024-05-06T00:27:50.549 INFO:teuthology.orchestra.run.smithi186.stdout:2575 read 0x7588a00 thru 0x758d4d0 (0x4ad1 bytes) 2024-05-06T00:27:50.555 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:50.551+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:50.580 INFO:teuthology.orchestra.run.smithi186.stdout:2576 write 0xd9fa600 thru 0xda012b3 (0x6cb4 bytes) 2024-05-06T00:27:50.583 INFO:teuthology.orchestra.run.smithi186.stdout:2577 punch from 0x3af5400 to 0x3b05200, (0xfe00 bytes) 2024-05-06T00:27:50.583 INFO:teuthology.orchestra.run.smithi186.stdout:2579 read 0x693b200 thru 0x69416fc (0x64fd bytes) 2024-05-06T00:27:50.584 INFO:teuthology.orchestra.run.smithi186.stdout:2580 read 0x217ae00 thru 0x217fb40 (0x4d41 bytes) 2024-05-06T00:27:50.584 INFO:teuthology.orchestra.run.smithi186.stdout:2582 read 0x4229000 thru 0x4234a17 (0xba18 bytes) 2024-05-06T00:27:50.584 INFO:teuthology.orchestra.run.smithi186.stdout:2583 punch from 0xb980600 to 0xb985200, (0x4c00 bytes) 2024-05-06T00:27:50.585 INFO:teuthology.orchestra.run.smithi186.stdout:2585 write 0x4b6d000 thru 0x4b72776 (0x5777 bytes) 2024-05-06T00:27:50.585 INFO:teuthology.orchestra.run.smithi186.stdout:2586 read 0x7575800 thru 0x757e07b (0x887c bytes) 2024-05-06T00:27:50.585 INFO:teuthology.orchestra.run.smithi186.stdout:2591 write 0xe25c00 thru 0xe2c36d (0x676e bytes) 2024-05-06T00:27:50.585 INFO:teuthology.orchestra.run.smithi186.stdout:2592 write 0x7e7c800 thru 0x7e83224 (0x6a25 bytes) 2024-05-06T00:27:50.585 INFO:teuthology.orchestra.run.smithi186.stdout:2593 read 0xac9e600 thru 0xaca1905 (0x3306 bytes) 2024-05-06T00:27:50.616 INFO:teuthology.orchestra.run.smithi186.stdout:2597 write 0xea2d200 thru 0xea3ae31 (0xdc32 bytes) 2024-05-06T00:27:50.624 INFO:teuthology.orchestra.run.smithi186.stdout:2598 trunc from 0xea3b000 to 0xb250400 2024-05-06T00:27:50.634 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:50.634 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:50.634 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:50.634 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:50.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:50.654 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.0. 2024-05-06T00:27:50.662 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:50.658+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:50.758 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:50.758+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:50.818 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:50.819 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:50.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:50.877 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:50.875+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:50.959 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:889 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3735552~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944734014000002, 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044456221999999997, 2024-05-06T00:27:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 889 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012238+0000", 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012239+0000", 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6000000000000003e-07 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012243+0000", 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5199999999999999e-06 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012253+0000", 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4620000000000002e-06 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012260+0000", 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4529999999999996e-06 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056325+0000", 2024-05-06T00:27:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044065026 2024-05-06T00:27:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056386+0000", 2024-05-06T00:27:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0158999999999998e-05 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056695+0000", 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030894199999999998 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:890 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3932160~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944701375000001, 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045011909000000003, 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 890 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012271+0000", 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012272+0000", 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7000000000000003e-07 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012276+0000", 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5430000000000004e-06 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012297+0000", 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0191999999999999e-05 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012309+0000", 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2354000000000001e-05 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056761+0000", 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044451630999999998 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.056861+0000", 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100241 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057283+0000", 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042207799999999999 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:891 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 4390912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944640630999999, 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045327649999999997, 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 891 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012332+0000", 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012333+0000", 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-07 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012338+0000", 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5639999999999996e-06 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012376+0000", 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7608000000000003e-05 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012401+0000", 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4752999999999998e-05 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057301+0000", 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044900714000000001 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057358+0000", 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6935000000000003e-05 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057659+0000", 2024-05-06T00:27:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030109599999999999 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:913 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 2457600~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944545573999999, 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045143827999999997, 2024-05-06T00:27:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 913 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012427+0000", 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012428+0000", 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7540000000000001e-06 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012446+0000", 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7492000000000001e-05 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012464+0000", 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8267e-05 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012479+0000", 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4484999999999999e-05 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057352+0000", 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044873203 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057450+0000", 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7621999999999995e-05 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057571+0000", 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121005 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:914 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944474526, 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045379520999999999, 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 914 2024-05-06T00:27:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012498+0000", 2024-05-06T00:27:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012499+0000", 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1799999999999999e-06 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012504+0000", 2024-05-06T00:27:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2599999999999996e-06 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012522+0000", 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7382999999999999e-05 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012539+0000", 2024-05-06T00:27:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7849000000000001e-05 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057642+0000", 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045102439000000001 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057756+0000", 2024-05-06T00:27:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114572 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057877+0000", 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120838 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:915 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944410783999999, 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045460430000000003, 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 915 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012562+0000", 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012563+0000", 2024-05-06T00:27:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0359999999999999e-06 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012568+0000", 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4500000000000003e-06 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012585+0000", 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7462999999999999e-05 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012602+0000", 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6385999999999999e-05 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057895+0000", 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045292978999999997 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.057949+0000", 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292999999999997e-05 2024-05-06T00:27:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058022+0000", 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2823000000000004e-05 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:916 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3309568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944276605999999, 2024-05-06T00:27:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045666393, 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 916 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012696+0000", 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012697+0000", 2024-05-06T00:27:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2100000000000001e-06 2024-05-06T00:27:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012703+0000", 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8869999999999997e-06 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012720+0000", 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7073000000000001e-05 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012733+0000", 2024-05-06T00:27:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3589e-05 2024-05-06T00:27:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058086+0000", 2024-05-06T00:27:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045352291000000003 2024-05-06T00:27:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058205+0000", 2024-05-06T00:27:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118794 2024-05-06T00:27:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058362+0000", 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015754899999999999 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:917 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3768320~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944219020999999, 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045746415999999998, 2024-05-06T00:27:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 917 2024-05-06T00:27:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012753+0000", 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012755+0000", 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3340000000000001e-06 2024-05-06T00:27:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012761+0000", 2024-05-06T00:27:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9359999999999996e-06 2024-05-06T00:27:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012779+0000", 2024-05-06T00:27:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8125999999999999e-05 2024-05-06T00:27:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012792+0000", 2024-05-06T00:27:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.327e-05 2024-05-06T00:27:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058383+0000", 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045591499000000001 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058431+0000", 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7658999999999999e-05 2024-05-06T00:27:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058500+0000", 2024-05-06T00:27:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8591999999999997e-05 2024-05-06T00:27:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:918 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 3964928~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944161759, 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045976217999999999, 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 918 2024-05-06T00:27:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012811+0000", 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012812+0000", 2024-05-06T00:27:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2130000000000001e-06 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012819+0000", 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9630000000000001e-06 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012842+0000", 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3119999999999999e-05 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012856+0000", 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3932e-05 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058570+0000", 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045713812999999999 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058662+0000", 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2893000000000007e-05 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058787+0000", 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000124284 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:919 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 4423680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944097844000002, 2024-05-06T00:27:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046059062999999997, 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 919 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012874+0000", 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012876+0000", 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.24e-06 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012881+0000", 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3460000000000001e-06 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012899+0000", 2024-05-06T00:27:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7549000000000001e-05 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012919+0000", 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0499000000000001e-05 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058807+0000", 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045887887000000002 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058862+0000", 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5325999999999998e-05 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058934+0000", 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1216000000000004e-05 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:920 3.a 3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10 [read 4620288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.944034952999999, 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046322267, 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 920 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012937+0000", 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012939+0000", 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1799999999999999e-06 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012944+0000", 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4360000000000001e-06 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012962+0000", 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.783e-05 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012975+0000", 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3682999999999999e-05 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058988+0000", 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046012151000000001 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059133+0000", 2024-05-06T00:27:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145253 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059260+0000", 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012673400000000001 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:949 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 2457600~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.943978112, 2024-05-06T00:27:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046356374999999998, 2024-05-06T00:27:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 949 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012994+0000", 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.012995+0000", 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013021+0000", 2024-05-06T00:27:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5422000000000002e-05 2024-05-06T00:27:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013040+0000", 2024-05-06T00:27:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9012999999999999e-05 2024-05-06T00:27:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013054+0000", 2024-05-06T00:27:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4503e-05 2024-05-06T00:27:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.058955+0000", 2024-05-06T00:27:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045900747999999998 2024-05-06T00:27:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059006+0000", 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1073999999999999e-05 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059351+0000", 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000344481 2024-05-06T00:27:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:950 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.943897587999999, 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046810959999999999, 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 950 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013075+0000", 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013076+0000", 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2100000000000001e-06 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013082+0000", 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9569999999999999e-06 2024-05-06T00:27:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013104+0000", 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2371999999999999e-05 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013119+0000", 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4769999999999999e-05 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059409+0000", 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046290169999999999 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059505+0000", 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5970000000000004e-05 2024-05-06T00:27:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059886+0000", 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038051099999999999 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:951 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.943834425999999, 2024-05-06T00:27:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047109395999999998, 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 951 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013138+0000", 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013139+0000", 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.203e-06 2024-05-06T00:27:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013144+0000", 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2329999999999997e-06 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013164+0000", 2024-05-06T00:27:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0063000000000001e-05 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013178+0000", 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3906e-05 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059905+0000", 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046727168999999999 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.059963+0000", 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7352999999999998e-05 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060247+0000", 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028446899999999998 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:952 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3309568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.943776065000002, 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047577908000000002, 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 952 2024-05-06T00:27:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013196+0000", 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013197+0000", 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1829999999999999e-06 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013203+0000", 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5400000000000003e-06 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013224+0000", 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0993e-05 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013238+0000", 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3553000000000001e-05 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060310+0000", 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047072307000000001 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060409+0000", 2024-05-06T00:27:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8793000000000001e-05 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060774+0000", 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036553900000000001 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:953 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3768320~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.943713363000001, 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047905053000000003, 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 953 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013259+0000", 2024-05-06T00:27:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013260+0000", 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.096e-06 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013302+0000", 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1508999999999998e-05 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013325+0000", 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3926000000000001e-05 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013341+0000", 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5305999999999999e-05 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060795+0000", 2024-05-06T00:27:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047453844000000002 2024-05-06T00:27:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.060860+0000", 2024-05-06T00:27:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5749999999999999e-05 2024-05-06T00:27:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061164+0000", 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030362199999999997 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:954 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 3964928~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.943586191000001, 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048325185, 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 954 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013386+0000", 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013387+0000", 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4300000000000001e-07 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013392+0000", 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5669999999999996e-06 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013405+0000", 2024-05-06T00:27:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3873000000000001e-05 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013415+0000", 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2429999999999994e-06 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061227+0000", 2024-05-06T00:27:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047811922999999999 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061327+0000", 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100046 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061711+0000", 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038458999999999998 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:955 3.1a 3:5aad81f3:::rbd_data.110c5ae73073.0000000000000007:10 [read 4423680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.943546306000002, 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048668028000000002, 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 955 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013426+0000", 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013427+0000", 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013430+0000", 2024-05-06T00:27:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013439+0000", 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4769999999999992e-06 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013446+0000", 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1690000000000003e-06 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061731+0000", 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048284322999999997 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.061789+0000", 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8251999999999997e-05 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062094+0000", 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030498999999999999 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4998.0:963 3.1a 3:5aa5ce21:::rbd_object_map.110c6c438b03:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.943515967, 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050888098, 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4998", 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 963 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013456+0000", 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013457+0000", 2024-05-06T00:27:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6400000000000001e-07 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013462+0000", 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6330000000000004e-06 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013472+0000", 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.009e-05 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.013479+0000", 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4499999999999998e-06 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062158+0000", 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048678720000000002 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062268+0000", 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109966 2024-05-06T00:27:51.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.062780+0000", 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051193899999999999 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064025+0000", 2024-05-06T00:27:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012450479999999999 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064186+0000", 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016146199999999999 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064285+0000", 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9079999999999993e-05 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:28.064344+0000", 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8946000000000001e-05 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.2966954780000002, 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:27:51.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:27:51.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.126 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.127 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:51.127 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:50.955+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:51.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:51.173 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:51.171+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:51.231 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:51.230+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:51.305 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:27:51.305 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:51.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:51.332 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:51.334+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:51.433 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:51.434+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:51.443 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:51.444 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:51.444 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:51.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:51.534 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:51.531+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.117793914000003, 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.579 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:27:51.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.116991925000001, 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.584 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:27:51.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:27:51.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.116905471000003, 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:27:51.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:27:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.116832877999997, 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:51.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:27:51.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.600 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.116773027000001, 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:27:51.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.602 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:51.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.605 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.116697344000002, 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.606 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2967 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 159744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.109073356000003, 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026274390000000002, 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.611 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2967 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467898+0000", 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.046e-06 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467917+0000", 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467937+0000", 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9613999999999999e-05 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467961+0000", 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.419e-05 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.493935+0000", 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025973922 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494053+0000", 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011866099999999999 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494171+0000", 2024-05-06T00:27:51.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011806099999999999 2024-05-06T00:27:51.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.616 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.617 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.617 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:51.617 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.108991770999999, 2024-05-06T00:27:51.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:27:51.618 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.618 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.619 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:27:51.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.619 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:51.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:51.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:27:51.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:27:51.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:27:51.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:27:51.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:27:51.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:27:51.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:27:51.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:27:51.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:27:51.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:27:51.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:27:51.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:27:51.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:27:51.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:27:51.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.629 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.629 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.630 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:51.630 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.108937564000001, 2024-05-06T00:27:51.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:27:51.630 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:27:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:27:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:27:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:27:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.108885415000003, 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:27:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.642 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3040 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.642 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:51.642 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.107030092999999, 2024-05-06T00:27:51.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026298018999999999, 2024-05-06T00:27:51.642 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.642 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.642 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3040 2024-05-06T00:27:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469941+0000", 2024-05-06T00:27:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0979999999999999e-06 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469946+0000", 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0170000000000004e-06 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469963+0000", 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6541999999999998e-05 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469977+0000", 2024-05-06T00:27:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4015999999999999e-05 2024-05-06T00:27:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495954+0000", 2024-05-06T00:27:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025976902999999999 2024-05-06T00:27:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496101+0000", 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147529 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496238+0000", 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136914 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.1069277, 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:27:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:27:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:27:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.10685136, 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.291713652999999, 2024-05-06T00:27:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:27:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:27:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:27:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:27:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:27:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:27:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:51.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:27:51.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.291661212000001, 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:27:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.284641229000002, 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:27:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:27:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:27:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:27:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:27:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:27:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:27:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:27:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:27:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:27:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:27:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:27:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:27:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:27:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:27:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:27:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:27:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:27:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:27:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:27:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:27:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.284594517999999, 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:27:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:27:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:27:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:27:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:27:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:27:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:27:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:27:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:27:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:27:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:27:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:27:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:27:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:27:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:27:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:27:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:27:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:27:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:27:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:27:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:27:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.284264409999999, 2024-05-06T00:27:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:27:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:27:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:27:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:27:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:27:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:27:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:27:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:27:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:27:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:27:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:27:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:27:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.284236488000001, 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:27:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.273369598999999, 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:51.713 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:51.714 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:27:51.716 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:51.711+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:51.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:51.816 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:51.814+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:51.841 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:51.842 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:51.856 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:51.917 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:51.915+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:51.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:52.017 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:52.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:52.017 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:52.017 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:52.017 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:52.018 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:52.018+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:52.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:52.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:52.080 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:52.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:52.097 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:52.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:52.114 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:52.118 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:52.115+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:52.134 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:52.155 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:52.175 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:52.175 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:52.190 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:52.190 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:52.204 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:52.219 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:52.215+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:52.320 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:52.322+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:52.360 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:52.360 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:52.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:52.465 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:52.466+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:52.522 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:52.522+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:52.539 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:52.540 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:52.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:27:52.624 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:52.619+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:52.647 INFO:teuthology.orchestra.run.smithi186.stdout:2599 write 0x340dc00 thru 0x3410767 (0x2b68 bytes) 2024-05-06T00:27:52.651 INFO:teuthology.orchestra.run.smithi186.stdout:2600 read 0x39e600 thru 0x3ae3c9 (0xfdca bytes) 2024-05-06T00:27:52.653 INFO:teuthology.orchestra.run.smithi186.stdout:2602 read 0x1704a00 thru 0x170d47a (0x8a7b bytes) 2024-05-06T00:27:52.653 INFO:teuthology.orchestra.run.smithi186.stdout:2604 read 0x2671800 thru 0x26813bc (0xfbbd bytes) 2024-05-06T00:27:52.654 INFO:teuthology.orchestra.run.smithi186.stdout:2605 read 0x7854600 thru 0x785af0b (0x690c bytes) 2024-05-06T00:27:52.654 INFO:teuthology.orchestra.run.smithi186.stdout:2606 read 0xa549400 thru 0xa54b5c1 (0x21c2 bytes) 2024-05-06T00:27:52.671 INFO:teuthology.orchestra.run.smithi186.stdout:2607 trunc from 0xb250400 to 0x4fe2600 2024-05-06T00:27:52.710 INFO:teuthology.orchestra.run.smithi186.stdout:2608 read 0x2dd8000 thru 0x2de4fb6 (0xcfb7 bytes) 2024-05-06T00:27:52.720 INFO:teuthology.orchestra.run.smithi186.stdout:2609 trunc from 0x4fe2600 to 0x2a59600 2024-05-06T00:27:52.724 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:52.719+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:52.749 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:52.749 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:52.768 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:52.772 INFO:teuthology.orchestra.run.smithi186.stdout:2611 write 0x4f18000 thru 0x4f192b6 (0x12b7 bytes) 2024-05-06T00:27:52.789 INFO:teuthology.orchestra.run.smithi186.stdout:2614 trunc from 0x4f19400 to 0xd923600 2024-05-06T00:27:52.810 INFO:teuthology.orchestra.run.smithi186.stdout:2615 write 0x9845400 thru 0x984855e (0x315f bytes) 2024-05-06T00:27:52.811 INFO:teuthology.orchestra.run.smithi186.stdout:2618 read 0x24b6400 thru 0x24b8f5e (0x2b5f bytes) 2024-05-06T00:27:52.811 INFO:teuthology.orchestra.run.smithi186.stdout:2619 punch from 0x4488a00 to 0x448ba00, (0x3000 bytes) 2024-05-06T00:27:52.811 INFO:teuthology.orchestra.run.smithi186.stdout:2620 read 0x439d400 thru 0x43a9a61 (0xc662 bytes) 2024-05-06T00:27:52.812 INFO:teuthology.orchestra.run.smithi186.stdout:2621 read 0xc54f600 thru 0xc55f147 (0xfb48 bytes) 2024-05-06T00:27:52.816 INFO:teuthology.orchestra.run.smithi186.stdout:2622 trunc from 0xd923600 to 0xb21ae00 2024-05-06T00:27:52.825 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:52.823+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:52.836 INFO:teuthology.orchestra.run.smithi186.stdout:2623 read 0x7e9b600 thru 0x7ea0548 (0x4f49 bytes) 2024-05-06T00:27:52.838 INFO:teuthology.orchestra.run.smithi186.stdout:2624 read 0x9972600 thru 0x997e562 (0xbf63 bytes) 2024-05-06T00:27:52.859 INFO:teuthology.orchestra.run.smithi186.stdout:2626 write 0xe954c00 thru 0xe95eb35 (0x9f36 bytes) 2024-05-06T00:27:52.860 INFO:teuthology.orchestra.run.smithi186.stdout:2627 read 0x30fd000 thru 0x31098c5 (0xc8c6 bytes) 2024-05-06T00:27:52.860 INFO:teuthology.orchestra.run.smithi186.stdout:2629 write 0x5112200 thru 0x5115e3b (0x3c3c bytes) 2024-05-06T00:27:52.860 INFO:teuthology.orchestra.run.smithi186.stdout:2632 read 0x7b56400 thru 0x7b582d4 (0x1ed5 bytes) 2024-05-06T00:27:52.865 INFO:teuthology.orchestra.run.smithi186.stdout:2633 trunc from 0xe95ec00 to 0xb63cc00 2024-05-06T00:27:52.893 INFO:teuthology.orchestra.run.smithi186.stdout:2636 write 0x9df4400 thru 0x9dfd8c7 (0x94c8 bytes) 2024-05-06T00:27:52.895 INFO:teuthology.orchestra.run.smithi186.stdout:2639 flatten 2024-05-06T00:27:52.908 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:52.908 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:52.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:52.926 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:52.926+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:52.930 INFO:teuthology.orchestra.run.smithi186.stdout:2642 write 0x60f2a00 thru 0x60fee4c (0xc44d bytes) 2024-05-06T00:27:52.954 INFO:teuthology.orchestra.run.smithi186.stdout:2643 trunc from 0xb63cc00 to 0x101e200 2024-05-06T00:27:52.985 INFO:teuthology.orchestra.run.smithi186.stdout:2644 punch from 0x1019a00 to 0x101e000, (0x4600 bytes) 2024-05-06T00:27:52.989 INFO:teuthology.orchestra.run.smithi186.stdout:2647 trunc from 0x101e200 to 0x507a00 2024-05-06T00:27:53.027 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:53.023+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:53.047 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:53.048 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:53.063 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:53.068 INFO:teuthology.orchestra.run.smithi186.stdout:2650 write 0xbfbd400 thru 0xbfbe8cf (0x14d0 bytes) 2024-05-06T00:27:53.070 INFO:teuthology.orchestra.run.smithi186.stdout:2651 read 0xadd8a00 thru 0xade2c0f (0xa210 bytes) 2024-05-06T00:27:53.070 INFO:teuthology.orchestra.run.smithi186.stdout:2656 read 0xbb2c00 thru 0xbbd796 (0xab97 bytes) 2024-05-06T00:27:53.070 INFO:teuthology.orchestra.run.smithi186.stdout:2657 read 0x19b2000 thru 0x19b936d (0x736e bytes) 2024-05-06T00:27:53.071 INFO:teuthology.orchestra.run.smithi186.stdout:2658 read 0x6af4200 thru 0x6af49a6 (0x7a7 bytes) 2024-05-06T00:27:53.074 INFO:teuthology.orchestra.run.smithi186.stdout:2659 trunc from 0xbfbea00 to 0xa686600 2024-05-06T00:27:53.107 INFO:teuthology.orchestra.run.smithi186.stdout:2661 write 0x1953800 thru 0x1959e6c (0x666d bytes) 2024-05-06T00:27:53.109 INFO:teuthology.orchestra.run.smithi186.stdout:2666 trunc from 0xa686600 to 0xac02000 2024-05-06T00:27:53.128 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:53.126+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:53.133 INFO:teuthology.orchestra.run.smithi186.stdout:2667 write 0x7726200 thru 0x772c762 (0x6563 bytes) 2024-05-06T00:27:53.163 INFO:teuthology.orchestra.run.smithi186.stdout:2668 write 0xbe64400 thru 0xbe6bba9 (0x77aa bytes) 2024-05-06T00:27:53.165 INFO:teuthology.orchestra.run.smithi186.stdout:2669 write 0x4f74a00 thru 0x4f7aacc (0x60cd bytes) 2024-05-06T00:27:53.165 INFO:teuthology.orchestra.run.smithi186.stdout:2670 read 0xb1f200 thru 0xb26c22 (0x7a23 bytes) 2024-05-06T00:27:53.170 INFO:teuthology.orchestra.run.smithi186.stdout:2671 trunc from 0xbe6bc00 to 0xe347c00 2024-05-06T00:27:53.196 INFO:teuthology.orchestra.run.smithi186.stdout:2675 read 0xb64b800 thru 0xb65ae4d (0xf64e bytes) 2024-05-06T00:27:53.208 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:53.208 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:53.221 INFO:teuthology.orchestra.run.smithi186.stdout:2679 trunc from 0xe347c00 to 0x454d600 2024-05-06T00:27:53.223 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:53.246 INFO:teuthology.orchestra.run.smithi186.stdout:2680 read 0x1041800 thru 0x1048847 (0x7048 bytes) 2024-05-06T00:27:53.248 INFO:teuthology.orchestra.run.smithi186.stdout:2681 punch from 0x1b14000 to 0x1b20c00, (0xcc00 bytes) 2024-05-06T00:27:53.248 INFO:teuthology.orchestra.run.smithi186.stdout:2683 write 0x3c04c00 thru 0x3c10b17 (0xbf18 bytes) 2024-05-06T00:27:53.248 INFO:teuthology.orchestra.run.smithi186.stdout:2684 read 0x9c4200 thru 0x9ceaef (0xa8f0 bytes) 2024-05-06T00:27:53.248 INFO:teuthology.orchestra.run.smithi186.stdout:2687 read 0x100b400 thru 0x1012968 (0x7569 bytes) 2024-05-06T00:27:53.251 INFO:teuthology.orchestra.run.smithi186.stdout:2692 trunc from 0x454d600 to 0x58d5800 2024-05-06T00:27:53.265 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:53.266+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:53.280 INFO:teuthology.orchestra.run.smithi186.stdout:2693 write 0x67fd800 thru 0x6800990 (0x3191 bytes) 2024-05-06T00:27:53.315 INFO:teuthology.orchestra.run.smithi186.stdout:2694 write 0xc7c5400 thru 0xc7d52b4 (0xfeb5 bytes) 2024-05-06T00:27:53.316 INFO:teuthology.orchestra.run.smithi186.stdout:2696 read 0xaec200 thru 0xaf0340 (0x4141 bytes) 2024-05-06T00:27:53.316 INFO:teuthology.orchestra.run.smithi186.stdout:2697 read 0x2693600 thru 0x269ab6f (0x7570 bytes) 2024-05-06T00:27:53.316 INFO:teuthology.orchestra.run.smithi186.stdout:2698 read 0x9af9400 thru 0x9b013c2 (0x7fc3 bytes) 2024-05-06T00:27:53.317 INFO:teuthology.orchestra.run.smithi186.stdout:2699 write 0x2e3a000 thru 0x2e499f6 (0xf9f7 bytes) 2024-05-06T00:27:53.317 INFO:teuthology.orchestra.run.smithi186.stdout:2702 write 0x779ae00 thru 0x779d4c9 (0x26ca bytes) 2024-05-06T00:27:53.317 INFO:teuthology.orchestra.run.smithi186.stdout:2703 punch from 0x26a6e00 to 0x26ab600, (0x4800 bytes) 2024-05-06T00:27:53.329 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:53.327+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:53.344 INFO:teuthology.orchestra.run.smithi186.stdout:2707 write 0xd523c00 thru 0xd533865 (0xfc66 bytes) 2024-05-06T00:27:53.345 INFO:teuthology.orchestra.run.smithi186.stdout:2709 write 0x5154200 thru 0x515a29f (0x60a0 bytes) 2024-05-06T00:27:53.345 INFO:teuthology.orchestra.run.smithi186.stdout:2710 punch from 0xc03b600 to 0xc04a000, (0xea00 bytes) 2024-05-06T00:27:53.348 INFO:teuthology.orchestra.run.smithi186.stdout:2711 punch from 0x955d200 to 0x9564600, (0x7400 bytes) 2024-05-06T00:27:53.348 INFO:teuthology.orchestra.run.smithi186.stdout:2712 punch from 0x5b17000 to 0x5b20400, (0x9400 bytes) 2024-05-06T00:27:53.348 INFO:teuthology.orchestra.run.smithi186.stdout:2714 read 0x8d50600 thru 0x8d53867 (0x3268 bytes) 2024-05-06T00:27:53.348 INFO:teuthology.orchestra.run.smithi186.stdout:2719 punch from 0xc77ca00 to 0xc78b200, (0xe800 bytes) 2024-05-06T00:27:53.369 INFO:teuthology.orchestra.run.smithi186.stdout:2721 trunc from 0xd533a00 to 0x2399e00 2024-05-06T00:27:53.370 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:53.370 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:53.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:53.399 INFO:teuthology.orchestra.run.smithi186.stdout:2722 read 0x114d000 thru 0x115bd25 (0xed26 bytes) 2024-05-06T00:27:53.432 INFO:teuthology.orchestra.run.smithi186.stdout:2724 write 0x7e5fa00 thru 0x7e6c3ab (0xc9ac bytes) 2024-05-06T00:27:53.433 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:53.434+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:53.452 INFO:teuthology.orchestra.run.smithi186.stdout:2725 write 0x93c9600 thru 0x93ced7a (0x577b bytes) 2024-05-06T00:27:53.454 INFO:teuthology.orchestra.run.smithi186.stdout:2726 punch from 0x4657e00 to 0x4659c00, (0x1e00 bytes) 2024-05-06T00:27:53.455 INFO:teuthology.orchestra.run.smithi186.stdout:2727 read 0x7d4b400 thru 0x7d4df96 (0x2b97 bytes) 2024-05-06T00:27:53.455 INFO:teuthology.orchestra.run.smithi186.stdout:2728 write 0x7f5200 thru 0x7f591a (0x71b bytes) 2024-05-06T00:27:53.455 INFO:teuthology.orchestra.run.smithi186.stdout:2729 read 0x6ab6600 thru 0x6ac1553 (0xaf54 bytes) 2024-05-06T00:27:53.455 INFO:teuthology.orchestra.run.smithi186.stdout:2730 read 0x23b8800 thru 0x23bce95 (0x4696 bytes) 2024-05-06T00:27:53.455 INFO:teuthology.orchestra.run.smithi186.stdout:2732 read 0x5a81600 thru 0x5a8728f (0x5c90 bytes) 2024-05-06T00:27:53.455 INFO:teuthology.orchestra.run.smithi186.stdout:2735 read 0x2c42400 thru 0x2c48b20 (0x6721 bytes) 2024-05-06T00:27:53.456 INFO:teuthology.orchestra.run.smithi186.stdout:2736 punch from 0x7e8600 to 0x7f3600, (0xb000 bytes) 2024-05-06T00:27:53.456 INFO:teuthology.orchestra.run.smithi186.stdout:2738 write 0x919ca00 thru 0x91a9ae8 (0xd0e9 bytes) 2024-05-06T00:27:53.456 INFO:teuthology.orchestra.run.smithi186.stdout:2739 punch from 0x47aea00 to 0x47b7a00, (0x9000 bytes) 2024-05-06T00:27:53.456 INFO:teuthology.orchestra.run.smithi186.stdout:2740 read 0x4a31a00 thru 0x4a3d1a2 (0xb7a3 bytes) 2024-05-06T00:27:53.456 INFO:teuthology.orchestra.run.smithi186.stdout:2743 write 0x7e68c00 thru 0x7e6a32d (0x172e bytes) 2024-05-06T00:27:53.484 INFO:teuthology.orchestra.run.smithi186.stdout:2744 write 0x9839a00 thru 0x98436c3 (0x9cc4 bytes) 2024-05-06T00:27:53.485 INFO:teuthology.orchestra.run.smithi186.stdout:2745 read 0x3fdac00 thru 0x3fe037a (0x577b bytes) 2024-05-06T00:27:53.485 INFO:teuthology.orchestra.run.smithi186.stdout:2747 read 0x525fc00 thru 0x5266acd (0x6ece bytes) 2024-05-06T00:27:53.490 INFO:teuthology.orchestra.run.smithi186.stdout:2749 trunc from 0x9843800 to 0x7df5a00 2024-05-06T00:27:53.534 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:53.531+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:53.544 INFO:teuthology.orchestra.run.smithi186.stdout:2750 trunc from 0x7df5a00 to 0xde2a00 2024-05-06T00:27:53.568 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:53.568 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:53.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:53.612 INFO:teuthology.orchestra.run.smithi186.stdout:2751 trunc from 0xde2a00 to 0xb7b3400 2024-05-06T00:27:53.636 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:53.634+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:53.638 INFO:teuthology.orchestra.run.smithi186.stdout:2753 read 0x586ca00 thru 0x5879aae (0xd0af bytes) 2024-05-06T00:27:53.640 INFO:teuthology.orchestra.run.smithi186.stdout:2755 write 0x9641c00 thru 0x96495bb (0x79bc bytes) 2024-05-06T00:27:53.640 INFO:teuthology.orchestra.run.smithi186.stdout:2761 write 0x4ea1000 thru 0x4eadc63 (0xcc64 bytes) 2024-05-06T00:27:53.640 INFO:teuthology.orchestra.run.smithi186.stdout:2762 read 0x5ba4200 thru 0x5ba88d2 (0x46d3 bytes) 2024-05-06T00:27:53.640 INFO:teuthology.orchestra.run.smithi186.stdout:2764 write 0x25ffc00 thru 0x2607272 (0x7673 bytes) 2024-05-06T00:27:53.653 INFO:teuthology.orchestra.run.smithi186.stdout:2765 trunc from 0xb7b3400 to 0x4980600 2024-05-06T00:27:53.735 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:53.734+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:53.760 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:53.760 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:53.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:53.836 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:53.831+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:53.896 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:53.896 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:53.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:53.937 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:53.938+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:54.038 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:54.038+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:54.069 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:54.069 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:54.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:54.139 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:54.135+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:54.193 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:54.206 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:54.206 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:54.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:54.261 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:54.259+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:54.341 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:54.342+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:54.343 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:54.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:54.360 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:54.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:54.385 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:54.385 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:54.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:54.396 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:54.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:54.414 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:54.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:54.433 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:54.442 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:54.442+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:54.523 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:27:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:27:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:27:54.524 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:54.537 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:54.545 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:54.542+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:54.644 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:54.639+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:54.647 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.197127979999998, 2024-05-06T00:27:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:27:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.196325991000002, 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:27:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:27:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.196239536999997, 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:27:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.196166943999998, 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:27:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:27:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.196107093000002, 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:27:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.196031410000003, 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2967 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 159744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.188407421999997, 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026274390000000002, 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2967 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467898+0000", 2024-05-06T00:27:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.046e-06 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467917+0000", 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467937+0000", 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9613999999999999e-05 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467961+0000", 2024-05-06T00:27:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.419e-05 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.493935+0000", 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025973922 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494053+0000", 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011866099999999999 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494171+0000", 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011806099999999999 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.188325837000001, 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:27:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.188271630000003, 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:27:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.188219480999997, 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:27:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:27:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3040 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.186364159, 2024-05-06T00:27:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026298018999999999, 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3040 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469941+0000", 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0979999999999999e-06 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469946+0000", 2024-05-06T00:27:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0170000000000004e-06 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469963+0000", 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6541999999999998e-05 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469977+0000", 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4015999999999999e-05 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495954+0000", 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025976902999999999 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496101+0000", 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147529 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496238+0000", 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136914 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.186261766000001, 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:27:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:27:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.186185426000002, 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:27:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.371047719, 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:27:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:27:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.370995277999999, 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:27:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:27:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:27:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.363975294999999, 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.363928584, 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:27:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:27:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:27:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.363598476, 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:27:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:27:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.363570553999999, 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:27:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:27:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:27:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.352703665, 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:27:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:27:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:27:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:27:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:27:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:27:54.756 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:27:54.757 INFO:teuthology.orchestra.run.smithi186.stdout:2766 trunc from 0x4980600 to 0x37d4400 2024-05-06T00:27:54.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:54.758 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:54.758+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:54.774 INFO:teuthology.orchestra.run.smithi186.stdout:2770 write 0xd241c00 thru 0xd24307c (0x147d bytes) 2024-05-06T00:27:54.775 INFO:teuthology.orchestra.run.smithi186.stdout:2773 punch from 0x3cbe200 to 0x3ccb400, (0xd200 bytes) 2024-05-06T00:27:54.807 INFO:teuthology.orchestra.run.smithi186.stdout:2775 write 0xe59e200 thru 0xe5a9635 (0xb436 bytes) 2024-05-06T00:27:54.809 INFO:teuthology.orchestra.run.smithi186.stdout:2776 read 0xd71e600 thru 0xd7228be (0x42bf bytes) 2024-05-06T00:27:54.810 INFO:teuthology.orchestra.run.smithi186.stdout:2777 punch from 0x28bb600 to 0x28be200, (0x2c00 bytes) 2024-05-06T00:27:54.814 INFO:teuthology.orchestra.run.smithi186.stdout:2778 trunc from 0xe5a9800 to 0xc284600 2024-05-06T00:27:54.847 INFO:teuthology.orchestra.run.smithi186.stdout:2779 punch from 0xb846a00 to 0xb84fe00, (0x9400 bytes) 2024-05-06T00:27:54.849 INFO:teuthology.orchestra.run.smithi186.stdout:2782 write 0x377fe00 thru 0x3787dd7 (0x7fd8 bytes) 2024-05-06T00:27:54.849 INFO:teuthology.orchestra.run.smithi186.stdout:2783 read 0x2e8e400 thru 0x2e9c30c (0xdf0d bytes) 2024-05-06T00:27:54.849 INFO:teuthology.orchestra.run.smithi186.stdout:2785 write 0x5889e00 thru 0x588c689 (0x288a bytes) 2024-05-06T00:27:54.849 INFO:teuthology.orchestra.run.smithi186.stdout:2787 read 0x5127200 thru 0x5133395 (0xc196 bytes) 2024-05-06T00:27:54.858 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:54.858+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:54.884 INFO:teuthology.orchestra.run.smithi186.stdout:2792 write 0xe560a00 thru 0xe56d2c9 (0xc8ca bytes) 2024-05-06T00:27:54.887 INFO:teuthology.orchestra.run.smithi186.stdout:2794 read 0x5362a00 thru 0x53703cc (0xd9cd bytes) 2024-05-06T00:27:54.888 INFO:teuthology.orchestra.run.smithi186.stdout:2801 write 0xfe1c00 thru 0xfe9d45 (0x8146 bytes) 2024-05-06T00:27:54.888 INFO:teuthology.orchestra.run.smithi186.stdout:2804 read 0x2a0c400 thru 0x2a16d97 (0xa998 bytes) 2024-05-06T00:27:54.899 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:54.900 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:54.914 INFO:teuthology.orchestra.run.smithi186.stdout:2805 write 0xe9b6600 thru 0xe9b80ec (0x1aed bytes) 2024-05-06T00:27:54.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:54.918 INFO:teuthology.orchestra.run.smithi186.stdout:2807 trunc from 0xe9b8200 to 0xe41ea00 2024-05-06T00:27:54.959 INFO:teuthology.orchestra.run.smithi186.stdout:2809 read 0x7278a00 thru 0x7286159 (0xd75a bytes) 2024-05-06T00:27:54.961 INFO:teuthology.orchestra.run.smithi186.stdout:2811 read 0x3e3b200 thru 0x3e40d43 (0x5b44 bytes) 2024-05-06T00:27:54.986 INFO:teuthology.orchestra.run.smithi186.stdout:2812 trunc from 0xe41ea00 to 0x551b400 2024-05-06T00:27:55.000 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:55.002+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:55.009 INFO:teuthology.orchestra.run.smithi186.stdout:2813 punch from 0x2c9c000 to 0x2c9d800, (0x1800 bytes) 2024-05-06T00:27:55.011 INFO:teuthology.orchestra.run.smithi186.stdout:2815 punch from 0x1fcb600 to 0x1fcfe00, (0x4800 bytes) 2024-05-06T00:27:55.012 INFO:teuthology.orchestra.run.smithi186.stdout:2818 write 0x2e63600 thru 0x2e67bf3 (0x45f4 bytes) 2024-05-06T00:27:55.012 INFO:teuthology.orchestra.run.smithi186.stdout:2820 punch from 0x1678600 to 0x1687000, (0xea00 bytes) 2024-05-06T00:27:55.039 INFO:teuthology.orchestra.run.smithi186.stdout:2822 write 0x86bd800 thru 0x86bff4b (0x274c bytes) 2024-05-06T00:27:55.047 INFO:teuthology.orchestra.run.smithi186.stdout:2824 trunc from 0x86c0000 to 0xae6d200 2024-05-06T00:27:55.060 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:55.055+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:55.069 INFO:teuthology.orchestra.run.smithi186.stdout:2826 punch from 0x4964a00 to 0x496ca00, (0x8000 bytes) 2024-05-06T00:27:55.078 INFO:teuthology.orchestra.run.smithi186.stdout:2828 trunc from 0xae6d200 to 0x7fcca00 2024-05-06T00:27:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:55.079 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:55.079 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:55.079 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:55.094 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:55.102 INFO:teuthology.orchestra.run.smithi186.stdout:2830 write 0x49e2400 thru 0x49e6b22 (0x4723 bytes) 2024-05-06T00:27:55.103 INFO:teuthology.orchestra.run.smithi186.stdout:2833 punch from 0xbeea00 to 0xbf4800, (0x5e00 bytes) 2024-05-06T00:27:55.104 INFO:teuthology.orchestra.run.smithi186.stdout:2834 punch from 0x2d24400 to 0x2d2d600, (0x9200 bytes) 2024-05-06T00:27:55.104 INFO:teuthology.orchestra.run.smithi186.stdout:2835 read 0xa95600 thru 0xa9ad74 (0x5775 bytes) 2024-05-06T00:27:55.104 INFO:teuthology.orchestra.run.smithi186.stdout:2842 read 0x24e2000 thru 0x24f1895 (0xf896 bytes) 2024-05-06T00:27:55.106 INFO:teuthology.orchestra.run.smithi186.stdout:2845 read 0x5204400 thru 0x520fa7d (0xb67e bytes) 2024-05-06T00:27:55.128 INFO:teuthology.orchestra.run.smithi186.stdout:2846 write 0x8680200 thru 0x8680ff4 (0xdf5 bytes) 2024-05-06T00:27:55.144 INFO:teuthology.orchestra.run.smithi186.stdout:2849 trunc from 0x8681000 to 0xdd7c600 2024-05-06T00:27:55.168 INFO:teuthology.orchestra.run.smithi186.stdout:2853 write 0x163000 thru 0x170d10 (0xdd11 bytes) 2024-05-06T00:27:55.169 INFO:teuthology.orchestra.run.smithi186.stdout:2855 punch from 0xceec400 to 0xcef3e00, (0x7a00 bytes) 2024-05-06T00:27:55.170 INFO:teuthology.orchestra.run.smithi186.stdout:2856 write 0x407ee00 thru 0x4087b25 (0x8d26 bytes) 2024-05-06T00:27:55.170 INFO:teuthology.orchestra.run.smithi186.stdout:2858 punch from 0x4f85600 to 0x4f8e600, (0x9000 bytes) 2024-05-06T00:27:55.171 INFO:teuthology.orchestra.run.smithi186.stdout:2859 punch from 0x5e10200 to 0x5e16400, (0x6200 bytes) 2024-05-06T00:27:55.171 INFO:teuthology.orchestra.run.smithi186.stdout:2860 write 0xb2ac00 thru 0xb3016c (0x556d bytes) 2024-05-06T00:27:55.172 INFO:teuthology.orchestra.run.smithi186.stdout:2862 punch from 0xdbe7e00 to 0xdbeae00, (0x3000 bytes) 2024-05-06T00:27:55.172 INFO:teuthology.orchestra.run.smithi186.stdout:2863 read 0xbe5ac00 thru 0xbe64b0e (0x9f0f bytes) 2024-05-06T00:27:55.184 INFO:teuthology.orchestra.run.smithi186.stdout:2864 trunc from 0xdd7c600 to 0x8577400 2024-05-06T00:27:55.184 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:55.186+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:55.220 INFO:teuthology.orchestra.run.smithi186.stdout:2865 read 0x11e7000 thru 0x11e9395 (0x2396 bytes) 2024-05-06T00:27:55.246 INFO:teuthology.orchestra.run.smithi186.stdout:2866 write 0xcfbf800 thru 0xcfca97e (0xb17f bytes) 2024-05-06T00:27:55.247 INFO:teuthology.orchestra.run.smithi186.stdout:2867 punch from 0x93bd600 to 0x93c8a00, (0xb400 bytes) 2024-05-06T00:27:55.248 INFO:teuthology.orchestra.run.smithi186.stdout:2871 clone 21 order 21 su 262144 sc 11 2024-05-06T00:27:55.262 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:55.259+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:55.274 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:55.274 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:55.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:55.274 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:55.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.274 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:55.274 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:55.274 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.470040736999998, 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:27:55.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:55.280 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.470024533999997, 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:55.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.286 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2527 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch ping cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.113930584999999, 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16552214100000001, 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:27:55.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2527 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160469+0000", 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160471+0000", 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.206e-06 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160477+0000", 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7440000000000001e-06 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160491+0000", 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3739e-05 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160506+0000", 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.473e-05 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160712+0000", 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020643899999999999 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.160736+0000", 2024-05-06T00:27:55.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3676000000000002e-05 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325517+0000", 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16478073700000001 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325685+0000", 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016879300000000001 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.325992+0000", 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030607699999999998 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:1976 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.117339597000001, 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050344819999999998, 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1976 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157060+0000", 2024-05-06T00:27:55.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157061+0000", 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3200000000000005e-07 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157064+0000", 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.022e-06 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157091+0000", 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6860999999999999e-05 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157101+0000", 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0293e-05 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207258+0000", 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050156559000000003 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207333+0000", 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5426000000000006e-05 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207405+0000", 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2026999999999993e-05 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.296 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:1977 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 874496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.117288308999999, 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050463471000000003, 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1977 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157112+0000", 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5400000000000001e-07 2024-05-06T00:27:55.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157116+0000", 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3720000000000001e-06 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157125+0000", 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5310000000000007e-06 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157132+0000", 2024-05-06T00:27:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.815e-06 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207435+0000", 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050303193000000003 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207484+0000", 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8646000000000001e-05 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207575+0000", 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1260000000000004e-05 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2109 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 465920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.117259062999999, 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050607512, 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2109 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157141+0000", 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.89e-07 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157144+0000", 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0340000000000001e-06 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157153+0000", 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2409999999999995e-06 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157159+0000", 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6719999999999998e-06 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207604+0000", 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050444298999999998 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207674+0000", 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0636000000000006e-05 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207748+0000", 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4140999999999994e-05 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2110 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.117232438999999, 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050757732999999999, 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2110 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157168+0000", 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:27:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157175+0000", 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5100000000000004e-06 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157183+0000", 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3429999999999996e-06 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157189+0000", 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1140000000000002e-06 2024-05-06T00:27:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207777+0000", 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050587886999999998 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207851+0000", 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4368000000000004e-05 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207925+0000", 2024-05-06T00:27:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4046000000000002e-05 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2111 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 486400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.117199855999999, 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050893031999999998, 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2111 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157200+0000", 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157201+0000", 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7899999999999998e-07 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157204+0000", 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0309999999999999e-06 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157212+0000", 2024-05-06T00:27:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6719999999999993e-06 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157219+0000", 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1750000000000002e-06 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.207953+0000", 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050734855000000002 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208022+0000", 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8468999999999998e-05 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208093+0000", 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1250999999999999e-05 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2112 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 496640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.117172854, 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051032028, 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2112 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157227+0000", 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157228+0000", 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8999999999999997e-07 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157230+0000", 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8049999999999998e-06 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157239+0000", 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2889999999999998e-06 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157245+0000", 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3389999999999998e-06 2024-05-06T00:27:55.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208121+0000", 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050875709999999998 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208189+0000", 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8711e-05 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208259+0000", 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9684000000000006e-05 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2113 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.117146649999999, 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051203918000000001, 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.319 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2113 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157253+0000", 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157655+0000", 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040158000000000001 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157661+0000", 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5840000000000002e-06 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157675+0000", 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4354e-05 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157684+0000", 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1859999999999998e-06 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208288+0000", 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050603883000000002 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208357+0000", 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8884e-05 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208457+0000", 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100447 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2114 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 517120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.323 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.116699249, 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050919006000000003, 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2114 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:55.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.157701+0000", 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158519+0000", 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00081808800000000002 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158524+0000", 2024-05-06T00:27:55.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.558e-06 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158538+0000", 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3703000000000001e-05 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158548+0000", 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0180000000000001e-05 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208486+0000", 2024-05-06T00:27:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049937466 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208551+0000", 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-05 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208620+0000", 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8281000000000006e-05 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2116 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 537600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.115802658, 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050356630999999999, 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2116 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158597+0000", 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158598+0000", 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7700000000000004e-07 2024-05-06T00:27:55.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158602+0000", 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4300000000000002e-06 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158610+0000", 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6510000000000007e-06 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158616+0000", 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1330000000000004e-06 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208813+0000", 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050196527999999997 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208881+0000", 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7718000000000005e-05 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208954+0000", 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3393999999999996e-05 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2117 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 547840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.115773597, 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050509317999999997, 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2117 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158626+0000", 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158627+0000", 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3200000000000005e-07 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158656+0000", 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8989999999999999e-05 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158666+0000", 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0093e-05 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158673+0000", 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3170000000000004e-06 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208987+0000", 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050313331000000003 2024-05-06T00:27:55.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209059+0000", 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2151999999999996e-05 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209136+0000", 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6803000000000003e-05 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2118 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 558080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.115714695000001, 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.05062726, 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2118 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158685+0000", 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158686+0000", 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-06T00:27:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158689+0000", 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0929999999999999e-06 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158697+0000", 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8560000000000007e-06 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158704+0000", 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8970000000000002e-06 2024-05-06T00:27:55.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209167+0000", 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050462662999999998 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209239+0000", 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2700999999999994e-05 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209313+0000", 2024-05-06T00:27:55.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3301999999999997e-05 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2119 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 568320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.115686348000001, 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050765355999999998, 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2119 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158714+0000", 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7899999999999998e-07 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158717+0000", 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1140000000000002e-06 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158726+0000", 2024-05-06T00:27:55.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2470000000000008e-06 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158735+0000", 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0429999999999996e-06 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209342+0000", 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050607632999999999 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209415+0000", 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2675999999999993e-05 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209479+0000", 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4064000000000002e-05 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2120 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 578560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.115655286999999, 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050922111999999999, 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2120 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158745+0000", 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158770+0000", 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4830000000000001e-05 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158774+0000", 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.656e-06 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158784+0000", 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3729999999999999e-06 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158791+0000", 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1860000000000003e-06 2024-05-06T00:27:55.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209512+0000", 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050721016000000001 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209593+0000", 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1629000000000001e-05 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209667+0000", 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3422000000000003e-05 2024-05-06T00:27:55.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2121 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 588800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.115598276, 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051033950000000002, 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2121 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.158802+0000", 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162924+0000", 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041224440000000003 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162931+0000", 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7909999999999999e-06 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162943+0000", 2024-05-06T00:27:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2009e-05 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.162952+0000", 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1160000000000005e-06 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209698+0000", 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046745791000000002 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209770+0000", 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2230000000000005e-05 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.209836+0000", 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5568999999999995e-05 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2149 3.12 3:4f250c41:::rbd_data.110ce7c6f85.0000000000000001:13 [read 875520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:55.354 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.110526580999998, 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050369018000000002, 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2149 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163873+0000", 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163874+0000", 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9699999999999996e-07 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163877+0000", 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.497e-06 2024-05-06T00:27:55.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163885+0000", 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0539999999999995e-06 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.163891+0000", 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2639999999999997e-06 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214165+0000", 2024-05-06T00:27:55.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.050274513999999999 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214202+0000", 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6390000000000002e-05 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.214242+0000", 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0701999999999998e-05 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4955.0:5833 3.c 3:361ba281:::rbd_data.110ce7c6f85.0000000000000048:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.613920001, 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.95602562599999996, 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4955", 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5833 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.359 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660480+0000", 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660487+0000", 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0979999999999996e-06 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660495+0000", 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2989999999999997e-06 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660514+0000", 2024-05-06T00:27:55.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8190999999999999e-05 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660526+0000", 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2892e-05 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660667+0000", 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014097599999999999 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660686+0000", 2024-05-06T00:27:55.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.889e-05 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:42.616398+0000", 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.95571166399999996 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:42.616506+0000", 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010761599999999999 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4955.0:5834 3.c 3:361ba281:::rbd_data.110ce7c6f85.0000000000000048:head [read 4096~28672] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.613815130000001, 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.95599441399999996, 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4955", 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5834 2024-05-06T00:27:55.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660585+0000", 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660586+0000", 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660595+0000", 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2969999999999992e-06 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660607+0000", 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1685000000000001e-05 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660615+0000", 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6610000000000006e-06 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660776+0000", 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016069399999999999 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:41.660795+0000", 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9398e-05 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:42.616529+0000", 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.95573370700000004 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:42.616579+0000", 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0142999999999997e-05 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.369 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:55.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:55.372 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:55.367+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:55.472 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:55.470+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:55.542 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:55.543 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:55.543 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:55.543 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:55.556 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:55.601 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:55.602+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:55.674 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:55.671+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:55.686 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4] out_osds: [5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:27:55.686 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-05-06T00:27:55.703 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:27:55.727 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:55.727 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:55.728 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:55.745 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:55.789 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:55.790+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:55.876 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:55.871+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:55.911 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2390 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 647168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.738298092000001, 2024-05-06T00:27:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051610458999999997, 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2390 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171462+0000", 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0100000000000004e-07 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171465+0000", 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0580000000000002e-06 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171472+0000", 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5989999999999996e-06 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171478+0000", 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8919999999999997e-06 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.222974+0000", 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051496171 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223025+0000", 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0377000000000002e-05 2024-05-06T00:27:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223071+0000", 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6560999999999997e-05 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2391 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.738123125000001, 2024-05-06T00:27:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051534153999999999, 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2391 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171637+0000", 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9800000000000003e-07 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171641+0000", 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2830000000000003e-06 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171651+0000", 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7219999999999994e-06 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171658+0000", 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1840000000000002e-06 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223088+0000", 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051430590999999998 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223127+0000", 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8921000000000002e-05 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223170+0000", 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2654999999999997e-05 2024-05-06T00:27:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2399 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 739328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.737077528, 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051514772, 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2399 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172682+0000", 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0800000000000004e-07 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172684+0000", 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3910000000000001e-06 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172691+0000", 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3999999999999997e-06 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172696+0000", 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5330000000000002e-06 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224101+0000", 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051404584000000003 2024-05-06T00:27:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224156+0000", 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4617000000000002e-05 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224196+0000", 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0639000000000002e-05 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2400 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 749568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.737054695000001, 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051590648000000003, 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2400 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172705+0000", 2024-05-06T00:27:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172707+0000", 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6350000000000002e-06 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172715+0000", 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4660000000000002e-06 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172720+0000", 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5249999999999996e-06 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224216+0000", 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051495763999999999 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224253+0000", 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6529999999999998e-05 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224295+0000", 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2190000000000001e-05 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2401 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 759808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.737031612999999, 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051663232000000003, 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2401 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172728+0000", 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8400000000000005e-07 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172730+0000", 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3010000000000001e-06 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172736+0000", 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-06 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172742+0000", 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5260000000000001e-06 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224320+0000", 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051578218000000002 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224357+0000", 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6418000000000003e-05 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224391+0000", 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3974999999999999e-05 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2402 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 770048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.737010263999998, 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051756008999999999, 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2402 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7100000000000001e-07 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172752+0000", 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.238e-06 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172758+0000", 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5989999999999998e-06 2024-05-06T00:27:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172764+0000", 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6030000000000004e-06 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224406+0000", 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051642527000000001 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224444+0000", 2024-05-06T00:27:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7251e-05 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224505+0000", 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1119999999999998e-05 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2403 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.736988683, 2024-05-06T00:27:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051822071999999997, 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2403 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172771+0000", 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9599999999999999e-07 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172791+0000", 2024-05-06T00:27:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9826e-05 2024-05-06T00:27:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172799+0000", 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2109999999999998e-06 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172806+0000", 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7479999999999996e-06 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224522+0000", 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051716744000000002 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224559+0000", 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6622999999999998e-05 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224592+0000", 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3423999999999997e-05 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2404 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 790528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.736939694, 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051892201999999998, 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2404 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172820+0000", 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1600000000000001e-07 2024-05-06T00:27:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172823+0000", 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7300000000000001e-06 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172831+0000", 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4120000000000001e-06 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172837+0000", 2024-05-06T00:27:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5260000000000001e-06 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224608+0000", 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051771073000000001 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224664+0000", 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5948e-05 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224711+0000", 2024-05-06T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7797000000000003e-05 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2405 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 800768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.736898177, 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051955289000000002, 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2405 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3099999999999998e-07 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172864+0000", 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8600000000000001e-06 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172872+0000", 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1049999999999995e-06 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172879+0000", 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4359999999999999e-06 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224728+0000", 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051849004999999997 2024-05-06T00:27:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224765+0000", 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7252000000000002e-05 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224816+0000", 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1100000000000002e-05 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2406 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 811008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.736869994999999, 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052018992, 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2406 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175329+0000", 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024402790000000001 2024-05-06T00:27:55.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175335+0000", 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9229999999999999e-06 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175344+0000", 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9029999999999993e-06 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175350+0000", 2024-05-06T00:27:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3430000000000001e-06 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224833+0000", 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049482544000000003 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224873+0000", 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9882999999999997e-05 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224908+0000", 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5117000000000003e-05 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4204 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.342435072000001, 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053179637000000002, 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4204 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567325+0000", 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8299999999999995e-07 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567329+0000", 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4730000000000002e-06 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567340+0000", 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.027e-05 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567348+0000", 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0199999999999994e-06 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614100+0000", 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046752557 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614266+0000", 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000165878 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614835+0000", 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00056892300000000004 2024-05-06T00:27:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619066+0000", 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042308729999999996 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620394+0000", 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013280080000000001 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620443+0000", 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9017000000000002e-05 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620504+0000", 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0735e-05 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4205 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.342394045999999, 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054249551999999999, 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4205 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567366+0000", 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567370+0000", 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2030000000000002e-06 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567396+0000", 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6274e-05 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567405+0000", 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7860000000000002e-06 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615526+0000", 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.048120586 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615603+0000", 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7948000000000007e-05 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615884+0000", 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028067199999999999 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620576+0000", 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046916049999999997 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621505+0000", 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00092975999999999996 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621553+0000", 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7555000000000001e-05 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621614+0000", 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1426000000000004e-05 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4227 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.333760769000001, 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054371322, 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4227 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575999+0000", 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.237e-06 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576007+0000", 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3660000000000003e-06 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576025+0000", 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7934000000000001e-05 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576041+0000", 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.607e-05 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619115+0000", 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043074610999999999 2024-05-06T00:27:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619242+0000", 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012649800000000001 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619799+0000", 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000557531 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627599+0000", 2024-05-06T00:27:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077998829999999996 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630216+0000", 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026171290000000002 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630304+0000", 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7564000000000004e-05 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630370+0000", 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5499000000000004e-05 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4228 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.333636786, 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054411922000000001, 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4228 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576124+0000", 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.33e-06 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576169+0000", 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5867e-05 2024-05-06T00:27:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576194+0000", 2024-05-06T00:27:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4536999999999999e-05 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576215+0000", 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0679999999999999e-05 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620628+0000", 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.044413312000000003 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620704+0000", 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6255999999999996e-05 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621011+0000", 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030724300000000001 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629112+0000", 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0081005490000000003 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630444+0000", 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00133248 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630492+0000", 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7144999999999997e-05 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630534+0000", 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2522999999999999e-05 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4229 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.333519922000001, 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054513172999999998, 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4229 2024-05-06T00:27:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576240+0000", 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4100000000000001e-06 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576272+0000", 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-05 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576294+0000", 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2283e-05 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576311+0000", 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6651e-05 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621671+0000", 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.045360079999999997 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621747+0000", 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5434999999999999e-05 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622136+0000", 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038980800000000001 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629125+0000", 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069886499999999999 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630609+0000", 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001483563 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630691+0000", 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2144999999999994e-05 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630752+0000", 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1518000000000003e-05 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4230 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.333425408, 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054593549999999998, 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4230 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576335+0000", 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5799999999999999e-06 2024-05-06T00:27:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576343+0000", 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5599999999999996e-06 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576362+0000", 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8810000000000001e-05 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576376+0000", 2024-05-06T00:27:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4887e-05 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622737+0000", 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046360459 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622836+0000", 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8911000000000002e-05 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.623379+0000", 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054296499999999996 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629139+0000", 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057601359999999999 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630805+0000", 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016661720000000001 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630878+0000", 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3348000000000003e-05 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630927+0000", 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8721999999999999e-05 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4231 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.333344691000001, 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054783287999999999, 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4231 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576416+0000", 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.28e-06 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576423+0000", 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576443+0000", 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.984e-05 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576459+0000", 2024-05-06T00:27:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6223e-05 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624037+0000", 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.047578143000000003 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624129+0000", 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1753000000000001e-05 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624431+0000", 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030202000000000001 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629149+0000", 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047174089999999997 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631083+0000", 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001934758 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631148+0000", 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4845999999999995e-05 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631198+0000", 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9443000000000001e-05 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4232 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.333279808, 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054906561, 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4232 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576481+0000", 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5269999999999999e-06 2024-05-06T00:27:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576488+0000", 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8129999999999996e-06 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576507+0000", 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8417e-05 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576523+0000", 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6546e-05 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.625281+0000", 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.04875753 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.625377+0000", 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6224000000000006e-05 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.626749+0000", 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001371585 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630390+0000", 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036414749999999999 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631265+0000", 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087439099999999997 2024-05-06T00:27:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631333+0000", 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8534000000000005e-05 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631386+0000", 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2519000000000003e-05 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4233 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.333215448000001, 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056371963999999997, 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4233 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576545+0000", 2024-05-06T00:27:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0669999999999999e-06 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576558+0000", 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3305999999999999e-05 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576576+0000", 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8587e-05 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576591+0000", 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4533e-05 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627664+0000", 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051072838000000002 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627956+0000", 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029239499999999998 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.628429+0000", 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047251999999999998 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632630+0000", 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042016800000000002 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632724+0000", 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3568999999999998e-05 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632839+0000", 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011482200000000001 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632915+0000", 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6647e-05 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4234 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.333150431, 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057621897999999998, 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4234 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576610+0000", 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.15e-06 2024-05-06T00:27:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576617+0000", 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8499999999999996e-06 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576636+0000", 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9205999999999999e-05 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576650+0000", 2024-05-06T00:27:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.416e-05 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629208+0000", 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052558219000000003 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629293+0000", 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4389999999999997e-05 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629620+0000", 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032745499999999999 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.633738+0000", 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004117725 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634045+0000", 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030709200000000002 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634146+0000", 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101362 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634230+0000", 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4289000000000006e-05 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.020 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:56.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:56.022 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:56.022+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:56.122 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:56.119+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:56.198 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:56.198 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:56.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5081.0:38 3.18 3.efeda9f8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:56.199028+0000", 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.00022347600000000001, 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026522299999999998, 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.199 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5081", 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1883189482", 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 38 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199028+0000", 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199028+0000", 2024-05-06T00:27:56.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199029+0000", 2024-05-06T00:27:56.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2869999999999999e-06 2024-05-06T00:27:56.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199037+0000", 2024-05-06T00:27:56.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4930000000000001e-06 2024-05-06T00:27:56.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199057+0000", 2024-05-06T00:27:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0409999999999999e-05 2024-05-06T00:27:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199072+0000", 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.501e-05 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199286+0000", 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021358299999999999 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5081.0:39 3.18 3.efeda9f8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-05-06T00:27:56.205 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:56.199097+0000", 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.00015410599999999999, 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030578199999999999, 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "queued for pg", 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5081", 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1883189482", 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199097+0000", 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199097+0000", 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199099+0000", 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3230000000000001e-06 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199109+0000", 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0363999999999999e-05 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199127+0000", 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7526000000000001e-05 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199145+0000", 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8726999999999999e-05 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5081.0:40 3.18 3.efeda9f8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:56.199167+0000", 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.4140000000000004e-05, 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032755000000000002, 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "queued for pg", 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5081", 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1883189482", 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.209 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199167+0000", 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199167+0000", 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199169+0000", 2024-05-06T00:27:56.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.254e-06 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199174+0000", 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7060000000000001e-06 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199192+0000", 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7776999999999999e-05 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199206+0000", 2024-05-06T00:27:56.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.433e-05 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5081.0:41 3.18 3.efeda9f8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:56.199224+0000", 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.7529999999999999e-05, 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035537299999999997, 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "queued for pg", 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5081", 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1883189482", 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 41 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199224+0000", 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199224+0000", 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199225+0000", 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199231+0000", 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5729999999999998e-06 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199248+0000", 2024-05-06T00:27:56.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7107000000000001e-05 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199262+0000", 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4250000000000001e-05 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5081.0:42 3.18 3.efeda9f8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:56.199281+0000", 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4294967295.9999709, 2024-05-06T00:27:56.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038462000000000001, 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "queued for pg", 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5081", 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1883189482", 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 42 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199281+0000", 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199281+0000", 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199282+0000", 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1069999999999999e-06 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199287+0000", 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0499999999999999e-06 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199305+0000", 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7929999999999999e-05 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199318+0000", 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3322999999999999e-05 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5081.0:43 3.18 3.efeda9f8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:56.199335+0000", 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4294967295.9999166, 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041811600000000001, 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "queued for pg", 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5081", 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1883189482", 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 43 2024-05-06T00:27:56.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199335+0000", 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199335+0000", 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199336+0000", 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1659999999999999e-06 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199341+0000", 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.95e-06 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199360+0000", 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9089999999999998e-05 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199374+0000", 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3837e-05 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5081.0:44 3.18 3.efeda9f8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:56.199391+0000", 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4294967295.9998603, 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044619199999999999, 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "queued for pg", 2024-05-06T00:27:56.222 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5081", 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1883189482", 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 44 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199391+0000", 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199391+0000", 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199392+0000", 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9999999999999995e-07 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199397+0000", 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7129999999999996e-06 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199415+0000", 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7612999999999999e-05 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:56.199428+0000", 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3713999999999999e-05 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:27:56.226 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 7 2024-05-06T00:27:56.226 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:56.227 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:56.228 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:56.223+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:56.328 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:56.327+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:56.349 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:56.349 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:56.353 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-05-06T00:27:56.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:56.429 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:56.427+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:56.437 INFO:teuthology.orchestra.run.smithi186.stdout:2873 read 0x3e74000 thru 0x3e79797 (0x5798 bytes) 2024-05-06T00:27:56.450 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:27:56.467 INFO:teuthology.orchestra.run.smithi186.stdout:2876 trunc from 0xcfcaa00 to 0x8267600 2024-05-06T00:27:56.496 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:56.496 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:56.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:56.496 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:56.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.497 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:860 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3407872~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.497 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:56.497 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.491124712000001, 2024-05-06T00:27:56.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054999124000000003, 2024-05-06T00:27:56.498 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 860 2024-05-06T00:27:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005693+0000", 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3969999999999999e-06 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005699+0000", 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1399999999999997e-06 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005718+0000", 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9190000000000001e-05 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005733+0000", 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4303e-05 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060250+0000", 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054516805000000002 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060353+0000", 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103509 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060691+0000", 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033777999999999998 2024-05-06T00:27:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:861 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.491064031000001, 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055437840000000002, 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 861 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005754+0000", 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005773+0000", 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9817e-05 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005792+0000", 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8770000000000002e-05 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005824+0000", 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1643000000000002e-05 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060742+0000", 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054918417999999997 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060845+0000", 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102507 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061190+0000", 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034541500000000001 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:862 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4063232~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490970554, 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055843853999999998, 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 862 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005847+0000", 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3200000000000001e-06 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005854+0000", 2024-05-06T00:27:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8299999999999998e-06 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005872+0000", 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.84e-05 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005886+0000", 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.347e-05 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061241+0000", 2024-05-06T00:27:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055355559999999998 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061343+0000", 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101335 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061690+0000", 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034693899999999999 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:863 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490910456999998, 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056056918999999997, 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 863 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005907+0000", 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.234e-06 2024-05-06T00:27:56.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005913+0000", 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8000000000000004e-06 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005932+0000", 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.895e-05 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005964+0000", 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2016e-05 2024-05-06T00:27:56.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061741+0000", 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055776824000000003 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061840+0000", 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9469000000000006e-05 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061963+0000", 2024-05-06T00:27:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000122626 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:899 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2392064~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490832337000001, 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056211016000000003, 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.517 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 899 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:56.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005985+0000", 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005992+0000", 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006011+0000", 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8669999999999999e-05 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006024+0000", 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-05 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062007+0000", 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055982453000000001 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062096+0000", 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9549000000000006e-05 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062195+0000", 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8999999999999994e-05 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:900 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2850816~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.521 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490772906, 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056426437000000003, 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 900 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006045+0000", 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.22e-06 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006064+0000", 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8997000000000001e-05 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006083+0000", 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9449999999999998e-05 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006098+0000", 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.451e-05 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062245+0000", 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056147619000000003 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062346+0000", 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100541 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062470+0000", 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012410000000000001 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:901 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490698825999999, 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056582556999999999, 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 901 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006119+0000", 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006125+0000", 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.207e-06 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006143+0000", 2024-05-06T00:27:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.821e-05 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006157+0000", 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4197e-05 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062513+0000", 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056355322999999999 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062600+0000", 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7060999999999999e-05 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062700+0000", 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100369 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:902 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490569385000001, 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056710658999999997, 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 902 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006249+0000", 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5939999999999999e-06 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006256+0000", 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1029999999999996e-06 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006277+0000", 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1637e-05 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006293+0000", 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.525e-05 2024-05-06T00:27:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062749+0000", 2024-05-06T00:27:56.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056456081999999998 2024-05-06T00:27:56.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062845+0000", 2024-05-06T00:27:56.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6194999999999996e-05 2024-05-06T00:27:56.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062958+0000", 2024-05-06T00:27:56.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000112798 2024-05-06T00:27:56.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.536 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:903 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3702784~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.49050166, 2024-05-06T00:27:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056894469000000003, 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 903 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006316+0000", 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2759999999999999e-06 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006322+0000", 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9839999999999998e-06 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006340+0000", 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8465999999999999e-05 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006354+0000", 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3137e-05 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063007+0000", 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056652965 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063100+0000", 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3638e-05 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063209+0000", 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109003 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:904 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4161536~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490442616999999, 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057092006000000001, 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 904 2024-05-06T00:27:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006375+0000", 2024-05-06T00:27:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006381+0000", 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0700000000000003e-06 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006399+0000", 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8372999999999998e-05 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006413+0000", 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3917000000000001e-05 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063258+0000", 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056844680000000002 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063352+0000", 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3991000000000002e-05 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063466+0000", 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113768 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:905 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4358144~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490384953, 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057200123999999998, 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 905 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006433+0000", 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006438+0000", 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5770000000000001e-06 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006456+0000", 2024-05-06T00:27:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006469+0000", 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2666999999999999e-05 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063509+0000", 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057039566999999999 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063563+0000", 2024-05-06T00:27:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4360000000000001e-05 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063632+0000", 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8663000000000003e-05 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:921 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490329699, 2024-05-06T00:27:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057645638999999999, 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 921 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006488+0000", 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0699999999999999e-06 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006493+0000", 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2159999999999997e-06 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006511+0000", 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8454e-05 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006525+0000", 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.342e-05 2024-05-06T00:27:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063673+0000", 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057148440000000002 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063765+0000", 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1608999999999997e-05 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064132+0000", 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036743000000000002 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:922 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2785280~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490274156000002, 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058081037000000002, 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 922 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006543+0000", 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.127e-06 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006568+0000", 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4777e-05 2024-05-06T00:27:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006588+0000", 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0203999999999998e-05 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006616+0000", 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7625999999999999e-05 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064169+0000", 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057553460000000001 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064264+0000", 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4773999999999998e-05 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064623+0000", 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000359069 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:923 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3244032~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490179022, 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058440913999999997, 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 923 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006639+0000", 2024-05-06T00:27:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006657+0000", 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8692999999999999e-05 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006677+0000", 2024-05-06T00:27:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9760999999999999e-05 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006692+0000", 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4467000000000001e-05 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064658+0000", 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057966692 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064746+0000", 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7851999999999998e-05 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065078+0000", 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033224200000000001 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.490105321000001, 2024-05-06T00:27:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:27:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:27:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.489876749, 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:27:56.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.489799465000001, 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:27:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.489603484, 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2006 3.d 3:b03dcdad:::rbd_data.110ce7c6f85.0000000000000002:13 [read 718848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.343180025999999, 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054771050000000002, 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2006 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153637+0000", 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1999999999999999e-07 2024-05-06T00:27:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153640+0000", 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.695e-06 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153647+0000", 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9619999999999996e-06 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153658+0000", 2024-05-06T00:27:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1249e-05 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208250+0000", 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054591955999999997 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208323+0000", 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-06T00:27:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208407+0000", 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4320999999999994e-05 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2007 3.d 3:b03dcdad:::rbd_data.110ce7c6f85.0000000000000002:13 [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.343150418, 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054911852999999997, 2024-05-06T00:27:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2007 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155145+0000", 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001478909 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155151+0000", 2024-05-06T00:27:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2840000000000003e-06 2024-05-06T00:27:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155166+0000", 2024-05-06T00:27:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4314000000000001e-05 2024-05-06T00:27:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155174+0000", 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3629999999999994e-06 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208436+0000", 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053262359000000002 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208505+0000", 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8677999999999995e-05 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208578+0000", 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2946000000000003e-05 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:56.595 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:56.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:56.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:56.599 INFO:teuthology.orchestra.run.smithi186.stdout:2877 write 0x769c600 thru 0x76a9ff5 (0xd9f6 bytes) 2024-05-06T00:27:56.599 INFO:teuthology.orchestra.run.smithi186.stdout:2878 write 0x3bb3e00 thru 0x3bc2094 (0xe295 bytes) 2024-05-06T00:27:56.599 INFO:teuthology.orchestra.run.smithi186.stdout:2880 punch from 0x64a1600 to 0x64aa400, (0x8e00 bytes) 2024-05-06T00:27:56.599 INFO:teuthology.orchestra.run.smithi186.stdout:2882 read 0x762400 thru 0x766b3f (0x4740 bytes) 2024-05-06T00:27:56.599 INFO:teuthology.orchestra.run.smithi186.stdout:2883 punch from 0x7af7000 to 0x7b05600, (0xe600 bytes) 2024-05-06T00:27:56.599 INFO:teuthology.orchestra.run.smithi186.stdout:2885 read 0x6d08c00 thru 0x6d124f3 (0x98f4 bytes) 2024-05-06T00:27:56.599 INFO:teuthology.orchestra.run.smithi186.stdout:2887 trunc from 0x8267600 to 0x123e400 2024-05-06T00:27:56.599 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:56.600 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:56.595+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:56.616 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:56.616 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:56.630 INFO:teuthology.orchestra.run.smithi186.stdout:2890 trunc from 0x123e400 to 0x7dd1200 2024-05-06T00:27:56.633 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:56.633 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:56.657 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:56.657 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:56.664 INFO:teuthology.orchestra.run.smithi186.stdout:2891 trunc from 0x7dd1200 to 0xea43a00 2024-05-06T00:27:56.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:56.678 INFO:teuthology.orchestra.run.smithi186.stdout:2892 write 0xda1400 thru 0xda3b90 (0x2791 bytes) 2024-05-06T00:27:56.697 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:56.695+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:56.703 INFO:teuthology.orchestra.run.smithi186.stdout:2893 trunc from 0xea43a00 to 0x1f02e00 2024-05-06T00:27:56.714 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.1 to 1 (10000) 2024-05-06T00:27:56.735 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:56.735 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:56.737 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-05-06T00:27:56.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:27:56.750 INFO:teuthology.orchestra.run.smithi186.stdout:2894 write 0x86a2200 thru 0x86a365e (0x145f bytes) 2024-05-06T00:27:56.759 INFO:teuthology.orchestra.run.smithi186.stdout:2895 trunc from 0x86a3800 to 0x32ca800 2024-05-06T00:27:56.780 INFO:teuthology.orchestra.run.smithi186.stdout:2897 read 0x21c7e00 thru 0x21d3824 (0xba25 bytes) 2024-05-06T00:27:56.782 INFO:teuthology.orchestra.run.smithi186.stdout:2898 read 0x1364a00 thru 0x1367b42 (0x3143 bytes) 2024-05-06T00:27:56.784 INFO:teuthology.orchestra.run.smithi186.stdout:2902 punch from 0x439000 to 0x446200, (0xd200 bytes) 2024-05-06T00:27:56.799 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:56.795+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:56.800 INFO:teuthology.orchestra.run.smithi186.stdout:2904 write 0x4347400 thru 0x43511a5 (0x9da6 bytes) 2024-05-06T00:27:56.800 INFO:teuthology.orchestra.run.smithi186.stdout:2906 write 0x12ac600 thru 0x12b20ae (0x5aaf bytes) 2024-05-06T00:27:56.820 INFO:teuthology.orchestra.run.smithi186.stdout:2907 write 0x5834000 thru 0x583d865 (0x9866 bytes) 2024-05-06T00:27:56.820 INFO:teuthology.orchestra.run.smithi186.stdout:2908 read 0x1e2400 thru 0x1ea418 (0x8019 bytes) 2024-05-06T00:27:56.821 INFO:teuthology.orchestra.run.smithi186.stdout:2912 write 0x17bd000 thru 0x17bea15 (0x1a16 bytes) 2024-05-06T00:27:56.821 INFO:teuthology.orchestra.run.smithi186.stdout:2916 punch from 0x537c800 to 0x5387800, (0xb000 bytes) 2024-05-06T00:27:56.822 INFO:teuthology.orchestra.run.smithi186.stdout:2917 read 0x254ec00 thru 0x255b56b (0xc96c bytes) 2024-05-06T00:27:56.822 INFO:teuthology.orchestra.run.smithi186.stdout:2918 punch from 0x107a800 to 0x107c600, (0x1e00 bytes) 2024-05-06T00:27:56.823 INFO:teuthology.orchestra.run.smithi186.stdout:2920 punch from 0x7d1800 to 0x7dfa00, (0xe200 bytes) 2024-05-06T00:27:56.823 INFO:teuthology.orchestra.run.smithi186.stdout:2921 write 0x35ad200 thru 0x35b3be1 (0x69e2 bytes) 2024-05-06T00:27:56.823 INFO:teuthology.orchestra.run.smithi186.stdout:2922 punch from 0x5813200 to 0x5818c00, (0x5a00 bytes) 2024-05-06T00:27:56.823 INFO:teuthology.orchestra.run.smithi186.stdout:2924 read 0x3916a00 thru 0x391f10d (0x870e bytes) 2024-05-06T00:27:56.836 INFO:teuthology.orchestra.run.smithi186.stdout:2926 trunc from 0x583da00 to 0xbebb400 2024-05-06T00:27:56.846 INFO:teuthology.orchestra.run.smithi186.stdout:2931 write 0x12c8200 thru 0x12ca5c7 (0x23c8 bytes) 2024-05-06T00:27:56.848 INFO:teuthology.orchestra.run.smithi186.stdout:2932 read 0x86dd600 thru 0x86e04e1 (0x2ee2 bytes) 2024-05-06T00:27:56.848 INFO:teuthology.orchestra.run.smithi186.stdout:2934 write 0x741fe00 thru 0x74222ab (0x24ac bytes) 2024-05-06T00:27:56.848 INFO:teuthology.orchestra.run.smithi186.stdout:2936 punch from 0x4fcd400 to 0x4fd6600, (0x9200 bytes) 2024-05-06T00:27:56.856 INFO:teuthology.orchestra.run.smithi186.stdout:2941 trunc from 0xbebb400 to 0x8145400 2024-05-06T00:27:56.871 INFO:teuthology.orchestra.run.smithi186.stdout:2944 write 0x6148a00 thru 0x6152069 (0x966a bytes) 2024-05-06T00:27:56.872 INFO:teuthology.orchestra.run.smithi186.stdout:2947 punch from 0x1ab6a00 to 0x1ac5800, (0xee00 bytes) 2024-05-06T00:27:56.872 INFO:teuthology.orchestra.run.smithi186.stdout:2950 punch from 0x7714c00 to 0x7718e00, (0x4200 bytes) 2024-05-06T00:27:56.872 INFO:teuthology.orchestra.run.smithi186.stdout:2951 read 0x845000 thru 0x84b268 (0x6269 bytes) 2024-05-06T00:27:56.874 INFO:teuthology.orchestra.run.smithi186.stdout:2953 read 0x312c000 thru 0x3138501 (0xc502 bytes) 2024-05-06T00:27:56.874 INFO:teuthology.orchestra.run.smithi186.stdout:2956 read 0x2172e00 thru 0x2177358 (0x4559 bytes) 2024-05-06T00:27:56.892 INFO:teuthology.orchestra.run.smithi186.stdout:2957 write 0x8785400 thru 0x8786808 (0x1409 bytes) 2024-05-06T00:27:56.893 INFO:teuthology.orchestra.run.smithi186.stdout:2958 write 0x162b200 thru 0x162b82d (0x62e bytes) 2024-05-06T00:27:56.894 INFO:teuthology.orchestra.run.smithi186.stdout:2959 read 0x7b92200 thru 0x7b931eb (0xfec bytes) 2024-05-06T00:27:56.894 INFO:teuthology.orchestra.run.smithi186.stdout:2960 punch from 0x80f9e00 to 0x80fc600, (0x2800 bytes) 2024-05-06T00:27:56.894 INFO:teuthology.orchestra.run.smithi186.stdout:2962 clone 22 order 18 su 1024 sc 6 2024-05-06T00:27:56.900 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:56.898+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:56.922 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:56.922 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:56.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:27:57.000 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:56.995+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:57.052 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:57.052 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:57.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:27:57.101 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:57.102+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:57.202 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:57.199+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:57.243 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:57.244 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:57.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:27:57.303 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:57.299+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:57.404 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:57.399+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:57.426 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:57.426 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:57.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:27:57.529 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:57.527+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:57.606 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:57.603+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:57.614 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:57.614 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:57.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:27:57.713 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:57.711+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:57.725 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.2 to 1 (10000) 2024-05-06T00:27:57.726 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:27:57.757 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-05-06T00:27:57.795 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:57.795 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:57.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:27:57.844 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone21 from 0x8786a00 (overlap 0x123e400) to 0xcc5200 2024-05-06T00:27:57.848 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:57.850+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:57.907 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:57.903+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:57.974 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:57.975 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:57.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:27:58.037 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:58.035+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:58.109 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:58.110+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:58.159 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:58.159 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:58.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:27:58.221 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:58.222+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:58.311 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:58.307+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:58.316 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:58.316 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:58.319 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-05-06T00:27:58.330 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:27:58.412 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:58.407+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:58.501 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:58.501 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:58.513 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:58.511+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:58.515 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:27:58.613 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:58.614+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:58.634 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:58.634 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:58.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:27:58.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:27:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.3 to 1 (10000) 2024-05-06T00:27:58.735 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:58.734+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:58.746 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:27:58.766 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-05-06T00:27:58.818 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:58.818+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:58.848 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:27:58.848 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:27:58.865 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:27:58.890 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:27:58.891 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:27:58.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:27:58.911 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:27:58.916 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:58.911+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:58.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:27:58.940 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:27:58.965 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:27:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:27:58.989 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:27:58.989 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:27:59.012 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:27:59.012 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:27:59.019 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:59.015+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:59.041 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:59.041 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:59.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:27:59.149 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:59.150+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:59.221 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:59.222+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:59.222 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:59.222 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:59.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:59.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:59.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.222 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2390 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 647168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.050845537000001, 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051610458999999997, 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2390 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171461+0000", 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171462+0000", 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0100000000000004e-07 2024-05-06T00:27:59.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171465+0000", 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0580000000000002e-06 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171472+0000", 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5989999999999996e-06 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171478+0000", 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8919999999999997e-06 2024-05-06T00:27:59.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.222974+0000", 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051496171 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223025+0000", 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0377000000000002e-05 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223071+0000", 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6560999999999997e-05 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2391 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.050670570000001, 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051534153999999999, 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.227 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2391 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171636+0000", 2024-05-06T00:27:59.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171637+0000", 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9800000000000003e-07 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171641+0000", 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2830000000000003e-06 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171651+0000", 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7219999999999994e-06 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.171658+0000", 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1840000000000002e-06 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223088+0000", 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051430590999999998 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223127+0000", 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8921000000000002e-05 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.223170+0000", 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2654999999999997e-05 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2399 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 739328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.231 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.049624973, 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051514772, 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2399 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172681+0000", 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172682+0000", 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0800000000000004e-07 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172684+0000", 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3910000000000001e-06 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172691+0000", 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3999999999999997e-06 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172696+0000", 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5330000000000002e-06 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224101+0000", 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051404584000000003 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224156+0000", 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4617000000000002e-05 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224196+0000", 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0639000000000002e-05 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.235 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2400 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 749568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.049602140000001, 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051590648000000003, 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2400 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172704+0000", 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172705+0000", 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172707+0000", 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6350000000000002e-06 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172715+0000", 2024-05-06T00:27:59.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4660000000000002e-06 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172720+0000", 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5249999999999996e-06 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224216+0000", 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051495763999999999 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224253+0000", 2024-05-06T00:27:59.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6529999999999998e-05 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224295+0000", 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2190000000000001e-05 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2401 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 759808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.049579057999999, 2024-05-06T00:27:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051663232000000003, 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2401 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172727+0000", 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172728+0000", 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8400000000000005e-07 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172730+0000", 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3010000000000001e-06 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172736+0000", 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-06 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172742+0000", 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5260000000000001e-06 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224320+0000", 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051578218000000002 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224357+0000", 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6418000000000003e-05 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224391+0000", 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3974999999999999e-05 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2402 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 770048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.049557708999998, 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051756008999999999, 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2402 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.245 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172749+0000", 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7100000000000001e-07 2024-05-06T00:27:59.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172752+0000", 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.238e-06 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172758+0000", 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5989999999999998e-06 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172764+0000", 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6030000000000004e-06 2024-05-06T00:27:59.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224406+0000", 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051642527000000001 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224444+0000", 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7251e-05 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224505+0000", 2024-05-06T00:27:59.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1119999999999998e-05 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2403 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.049536128, 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051822071999999997, 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2403 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172770+0000", 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172771+0000", 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9599999999999999e-07 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172791+0000", 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9826e-05 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172799+0000", 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2109999999999998e-06 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172806+0000", 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7479999999999996e-06 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224522+0000", 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051716744000000002 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224559+0000", 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6622999999999998e-05 2024-05-06T00:27:59.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224592+0000", 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3423999999999997e-05 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2404 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 790528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.049487139, 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051892201999999998, 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2404 2024-05-06T00:27:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172819+0000", 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172820+0000", 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1600000000000001e-07 2024-05-06T00:27:59.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172823+0000", 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7300000000000001e-06 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172831+0000", 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4120000000000001e-06 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172837+0000", 2024-05-06T00:27:59.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5260000000000001e-06 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224608+0000", 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051771073000000001 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224664+0000", 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5948e-05 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224711+0000", 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7797000000000003e-05 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2405 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 800768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.049445622, 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051955289000000002, 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.261 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2405 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:59.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172861+0000", 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3099999999999998e-07 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172864+0000", 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8600000000000001e-06 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172872+0000", 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1049999999999995e-06 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172879+0000", 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4359999999999999e-06 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224728+0000", 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051849004999999997 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224765+0000", 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7252000000000002e-05 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224816+0000", 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1100000000000002e-05 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2406 3.0 3:0771907f:::rbd_data.110ce7c6f85.0000000000000003:13 [read 811008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.049417439999999, 2024-05-06T00:27:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052018992, 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2406 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.172889+0000", 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175329+0000", 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024402790000000001 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175335+0000", 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9229999999999999e-06 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175344+0000", 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9029999999999993e-06 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.175350+0000", 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3430000000000001e-06 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224833+0000", 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049482544000000003 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224873+0000", 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9882999999999997e-05 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.224908+0000", 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5117000000000003e-05 2024-05-06T00:27:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4204 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.654982517000001, 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053179637000000002, 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4204 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567324+0000", 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567325+0000", 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8299999999999995e-07 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567329+0000", 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4730000000000002e-06 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567340+0000", 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.027e-05 2024-05-06T00:27:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567348+0000", 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0199999999999994e-06 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614100+0000", 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046752557 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614266+0000", 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000165878 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.614835+0000", 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00056892300000000004 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619066+0000", 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042308729999999996 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620394+0000", 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013280080000000001 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620443+0000", 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9017000000000002e-05 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620504+0000", 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0735e-05 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4205 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.654941490999999, 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054249551999999999, 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4205 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567365+0000", 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567366+0000", 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567370+0000", 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2030000000000002e-06 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567396+0000", 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6274e-05 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.567405+0000", 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7860000000000002e-06 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615526+0000", 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.048120586 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615603+0000", 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7948000000000007e-05 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.615884+0000", 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028067199999999999 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620576+0000", 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046916049999999997 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621505+0000", 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00092975999999999996 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621553+0000", 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7555000000000001e-05 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621614+0000", 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1426000000000004e-05 2024-05-06T00:27:59.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4227 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.646308214000001, 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054371322, 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4227 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575998+0000", 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.575999+0000", 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.237e-06 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576007+0000", 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3660000000000003e-06 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576025+0000", 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7934000000000001e-05 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576041+0000", 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.607e-05 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619115+0000", 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043074610999999999 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619242+0000", 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012649800000000001 2024-05-06T00:27:59.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.619799+0000", 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000557531 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627599+0000", 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077998829999999996 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630216+0000", 2024-05-06T00:27:59.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026171290000000002 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630304+0000", 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7564000000000004e-05 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630370+0000", 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5499000000000004e-05 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4228 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.646184230999999, 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054411922000000001, 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4228 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576122+0000", 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576124+0000", 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.33e-06 2024-05-06T00:27:59.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576169+0000", 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5867e-05 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576194+0000", 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4536999999999999e-05 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576215+0000", 2024-05-06T00:27:59.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0679999999999999e-05 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620628+0000", 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.044413312000000003 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.620704+0000", 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6255999999999996e-05 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621011+0000", 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030724300000000001 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629112+0000", 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0081005490000000003 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630444+0000", 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00133248 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630492+0000", 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7144999999999997e-05 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630534+0000", 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2522999999999999e-05 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4229 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:59.293 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.646067367000001, 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054513172999999998, 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4229 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:59.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576239+0000", 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576240+0000", 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4100000000000001e-06 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576272+0000", 2024-05-06T00:27:59.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-05 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576294+0000", 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2283e-05 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576311+0000", 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6651e-05 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621671+0000", 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.045360079999999997 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.621747+0000", 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5434999999999999e-05 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622136+0000", 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038980800000000001 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629125+0000", 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069886499999999999 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630609+0000", 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001483563 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630691+0000", 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2144999999999994e-05 2024-05-06T00:27:59.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630752+0000", 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1518000000000003e-05 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4230 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.645972853, 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054593549999999998, 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4230 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576334+0000", 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576335+0000", 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5799999999999999e-06 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576343+0000", 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5599999999999996e-06 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576362+0000", 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8810000000000001e-05 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576376+0000", 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4887e-05 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622737+0000", 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046360459 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.622836+0000", 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8911000000000002e-05 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.623379+0000", 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054296499999999996 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629139+0000", 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057601359999999999 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630805+0000", 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016661720000000001 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630878+0000", 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3348000000000003e-05 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630927+0000", 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8721999999999999e-05 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.304 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4231 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.645892136000001, 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054783287999999999, 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4231 2024-05-06T00:27:59.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576414+0000", 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576416+0000", 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.28e-06 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576423+0000", 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576443+0000", 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.984e-05 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576459+0000", 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6223e-05 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624037+0000", 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.047578143000000003 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624129+0000", 2024-05-06T00:27:59.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1753000000000001e-05 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.624431+0000", 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030202000000000001 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629149+0000", 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047174089999999997 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631083+0000", 2024-05-06T00:27:59.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001934758 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631148+0000", 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4845999999999995e-05 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631198+0000", 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9443000000000001e-05 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4232 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.645827253, 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054906561, 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4232 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576479+0000", 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576481+0000", 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5269999999999999e-06 2024-05-06T00:27:59.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576488+0000", 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8129999999999996e-06 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576507+0000", 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8417e-05 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576523+0000", 2024-05-06T00:27:59.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6546e-05 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.625281+0000", 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.04875753 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.625377+0000", 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6224000000000006e-05 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.626749+0000", 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001371585 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.630390+0000", 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036414749999999999 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631265+0000", 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087439099999999997 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631333+0000", 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8534000000000005e-05 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.631386+0000", 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2519000000000003e-05 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4233 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.645762893000001, 2024-05-06T00:27:59.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056371963999999997, 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4233 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576544+0000", 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576545+0000", 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0669999999999999e-06 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576558+0000", 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3305999999999999e-05 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576576+0000", 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8587e-05 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576591+0000", 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4533e-05 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627664+0000", 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.051072838000000002 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.627956+0000", 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029239499999999998 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.628429+0000", 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047251999999999998 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632630+0000", 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042016800000000002 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632724+0000", 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3568999999999998e-05 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632839+0000", 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011482200000000001 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.632915+0000", 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6647e-05 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:4234 3.0 3:0445f230:::rbd_object_map.110c427bac34:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.645697876, 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057621897999999998, 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4234 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576609+0000", 2024-05-06T00:27:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576610+0000", 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.15e-06 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576617+0000", 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8499999999999996e-06 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576636+0000", 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9205999999999999e-05 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.576650+0000", 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.416e-05 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629208+0000", 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052558219000000003 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629293+0000", 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4389999999999997e-05 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.629620+0000", 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032745499999999999 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.633738+0000", 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004117725 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:27:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634045+0000", 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030709200000000002 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634146+0000", 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101362 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.634230+0000", 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4289000000000006e-05 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.328 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:59.329 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:27:59.330 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:27:59.327+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:27:59.430 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:27:59.430+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:27:59.497 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:27:59.497 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:59.511 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:27:59.553 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:27:59.554+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:27:59.594 INFO:teuthology.orchestra.run.smithi186.stdout:2964 read 0x63d1600 thru 0x63d2689 (0x108a bytes) 2024-05-06T00:27:59.595 INFO:teuthology.orchestra.run.smithi186.stdout:2968 punch from 0x196a400 to 0x1971200, (0x6e00 bytes) 2024-05-06T00:27:59.631 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:27:59.630+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:27:59.692 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:59.692 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:27:59.692 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:27:59.692 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:27:59.692 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:59.707 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:27:59.732 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:59.727+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:59.736 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.4 to 1 (10000) 2024-05-06T00:27:59.760 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-05-06T00:27:59.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:27:59.765 INFO:teuthology.orchestra.run.smithi186.stdout:2970 write 0x8c65a00 thru 0x8c6ba7b (0x607c bytes) 2024-05-06T00:27:59.810 INFO:teuthology.orchestra.run.smithi186.stdout:2973 trunc from 0x8c6bc00 to 0x2413e00 2024-05-06T00:27:59.833 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:27:59.831+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:27:59.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:27:59.875 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:27:59.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:27:59.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:860 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3407872~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.869259473, 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054999124000000003, 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 860 2024-05-06T00:27:59.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005692+0000", 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005693+0000", 2024-05-06T00:27:59.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3969999999999999e-06 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005699+0000", 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1399999999999997e-06 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005718+0000", 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9190000000000001e-05 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005733+0000", 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4303e-05 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060250+0000", 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054516805000000002 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060353+0000", 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103509 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060691+0000", 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033777999999999998 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:861 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:59.880 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.869198791999999, 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055437840000000002, 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 861 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:59.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005752+0000", 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005754+0000", 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005773+0000", 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9817e-05 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005792+0000", 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8770000000000002e-05 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005824+0000", 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1643000000000002e-05 2024-05-06T00:27:59.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060742+0000", 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054918417999999997 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.060845+0000", 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102507 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061190+0000", 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034541500000000001 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:862 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4063232~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.869105314999999, 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055843853999999998, 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 862 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005846+0000", 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005847+0000", 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3200000000000001e-06 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005854+0000", 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8299999999999998e-06 2024-05-06T00:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005872+0000", 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.84e-05 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005886+0000", 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.347e-05 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061241+0000", 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055355559999999998 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061343+0000", 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101335 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061690+0000", 2024-05-06T00:27:59.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034693899999999999 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:863 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.869045218, 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056056918999999997, 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 863 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005906+0000", 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005907+0000", 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.234e-06 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005913+0000", 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8000000000000004e-06 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005932+0000", 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.895e-05 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005964+0000", 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2016e-05 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061741+0000", 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055776824000000003 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061840+0000", 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9469000000000006e-05 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.061963+0000", 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000122626 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.894 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:899 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2392064~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868967097999999, 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056211016000000003, 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 899 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005984+0000", 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005985+0000", 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-06T00:27:59.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.005992+0000", 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006011+0000", 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8669999999999999e-05 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006024+0000", 2024-05-06T00:27:59.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-05 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062007+0000", 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.055982453000000001 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062096+0000", 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9549000000000006e-05 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062195+0000", 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8999999999999994e-05 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:900 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 2850816~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868907666999998, 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056426437000000003, 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.899 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 900 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006044+0000", 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006045+0000", 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.22e-06 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006064+0000", 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8997000000000001e-05 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006083+0000", 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9449999999999998e-05 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006098+0000", 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.451e-05 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062245+0000", 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056147619000000003 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062346+0000", 2024-05-06T00:27:59.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100541 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062470+0000", 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012410000000000001 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:901 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868833587000001, 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056582556999999999, 2024-05-06T00:27:59.903 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 901 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006118+0000", 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006119+0000", 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006125+0000", 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.207e-06 2024-05-06T00:27:59.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006143+0000", 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.821e-05 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006157+0000", 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4197e-05 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062513+0000", 2024-05-06T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056355322999999999 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062600+0000", 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7060999999999999e-05 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062700+0000", 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100369 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:902 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868704145999999, 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056710658999999997, 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 902 2024-05-06T00:27:59.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006247+0000", 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006249+0000", 2024-05-06T00:27:59.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5939999999999999e-06 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006256+0000", 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1029999999999996e-06 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006277+0000", 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1637e-05 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006293+0000", 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.525e-05 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062749+0000", 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056456081999999998 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062845+0000", 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6194999999999996e-05 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.062958+0000", 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000112798 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:903 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 3702784~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868636421000001, 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056894469000000003, 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 903 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006315+0000", 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006316+0000", 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2759999999999999e-06 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006322+0000", 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9839999999999998e-06 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006340+0000", 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8465999999999999e-05 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006354+0000", 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3137e-05 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063007+0000", 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056652965 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063100+0000", 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3638e-05 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063209+0000", 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109003 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:904 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4161536~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868577378000001, 2024-05-06T00:27:59.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057092006000000001, 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 904 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006374+0000", 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006375+0000", 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006381+0000", 2024-05-06T00:27:59.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0700000000000003e-06 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006399+0000", 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8372999999999998e-05 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006413+0000", 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3917000000000001e-05 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063258+0000", 2024-05-06T00:27:59.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056844680000000002 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063352+0000", 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3991000000000002e-05 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063466+0000", 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113768 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:905 3.1e 3:7d52796b:::rbd_data.110c5ae73073.0000000000000000:10 [read 4358144~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868519714000001, 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057200123999999998, 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 905 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006431+0000", 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006433+0000", 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.19e-06 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006438+0000", 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5770000000000001e-06 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006456+0000", 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8099999999999999e-05 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006469+0000", 2024-05-06T00:27:59.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2666999999999999e-05 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063509+0000", 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057039566999999999 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063563+0000", 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4360000000000001e-05 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063632+0000", 2024-05-06T00:27:59.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8663000000000003e-05 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:921 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868464459999998, 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057645638999999999, 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.925 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 921 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006487+0000", 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006488+0000", 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0699999999999999e-06 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006493+0000", 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2159999999999997e-06 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006511+0000", 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8454e-05 2024-05-06T00:27:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006525+0000", 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.342e-05 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063673+0000", 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057148440000000002 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.063765+0000", 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1608999999999997e-05 2024-05-06T00:27:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064132+0000", 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036743000000000002 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:922 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 2785280~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868408917, 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058081037000000002, 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 922 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006542+0000", 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006543+0000", 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.127e-06 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006568+0000", 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4777e-05 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006588+0000", 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0203999999999998e-05 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006616+0000", 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7625999999999999e-05 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064169+0000", 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057553460000000001 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064264+0000", 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4773999999999998e-05 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064623+0000", 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000359069 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:923 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3244032~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868313783000001, 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058440913999999997, 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 923 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006637+0000", 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006639+0000", 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006657+0000", 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8692999999999999e-05 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006677+0000", 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9760999999999999e-05 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006692+0000", 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4467000000000001e-05 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064658+0000", 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057966692 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.064746+0000", 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7851999999999998e-05 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065078+0000", 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033224200000000001 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868240082, 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:27:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.868011509999999, 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:27:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.867934225999999, 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:27:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.867738244999998, 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:27:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:27:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:27:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2006 3.d 3:b03dcdad:::rbd_data.110ce7c6f85.0000000000000002:13 [read 718848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.721314787000001, 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054771050000000002, 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2006 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153636+0000", 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153637+0000", 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1999999999999999e-07 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153640+0000", 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.695e-06 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153647+0000", 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9619999999999996e-06 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153658+0000", 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1249e-05 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208250+0000", 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054591955999999997 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208323+0000", 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208407+0000", 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4320999999999994e-05 2024-05-06T00:27:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5133.0:2007 3.d 3:b03dcdad:::rbd_data.110ce7c6f85.0000000000000002:13 [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.721285178999999, 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054911852999999997, 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5133", 2024-05-06T00:27:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/693353247", 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2007 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.153666+0000", 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155145+0000", 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001478909 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155151+0000", 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2840000000000003e-06 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155166+0000", 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4314000000000001e-05 2024-05-06T00:27:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.155174+0000", 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3629999999999994e-06 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208436+0000", 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053262359000000002 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208505+0000", 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8677999999999995e-05 2024-05-06T00:27:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:37.208578+0000", 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2946000000000003e-05 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:27:59.964 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:27:59.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:27:59.966 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:27:59.963+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:27:59.997 INFO:teuthology.orchestra.run.smithi186.stdout:2979 read 0x7a4c00 thru 0x7abfc0 (0x73c1 bytes) 2024-05-06T00:28:00.030 INFO:teuthology.orchestra.run.smithi186.stdout:2980 write 0x3a91e00 thru 0x3a9b2d6 (0x94d7 bytes) 2024-05-06T00:28:00.066 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:00.063+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:00.087 INFO:teuthology.orchestra.run.smithi186.stdout:2981 write 0x6dd7a00 thru 0x6de0d3d (0x933e bytes) 2024-05-06T00:28:00.101 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:00.101 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:00.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:28:00.123 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:28:00.128 INFO:teuthology.orchestra.run.smithi186.stdout:2982 write 0x9807600 thru 0x980c896 (0x5297 bytes) 2024-05-06T00:28:00.137 INFO:teuthology.orchestra.run.smithi186.stdout:2985 trunc from 0x980ca00 to 0x5cc4c00 2024-05-06T00:28:00.142 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-05-06T00:27:59.738898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":627328807,"num_objects":1072,"num_object_clones":214,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1072,"num_whiteouts":538,"num_read":100426,"num_read_kb":401405,"num_write":9180,"num_write_kb":99559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":10280781,"num_keys_recovered":241,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8438,"ondisk_log_size":8438,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":219936,"kb_used_data":53624,"kb_used_omap":6,"kb_used_meta":166265,"kb_avail":377267424,"statfs":{"total":386547056640,"available":386321842176,"internally_reserved":0,"allocated":54910976,"data_stored":78064126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":170256150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41098404,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":6100,"num_read_kb":14703,"num_write":666,"num_write_kb":3389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":4464034,"num_keys_recovered":103,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.033840"},"pg_stats":[{"pgid":"3.1f","version":"74'200","reported_seq":3602,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.916383+0000","last_change":"2024-05-06T00:27:54.717030+0000","last_active":"2024-05-06T00:27:56.916383+0000","last_peered":"2024-05-06T00:27:56.916383+0000","last_clean":"2024-05-06T00:27:56.916383+0000","last_became_active":"2024-05-06T00:27:54.715397+0000","last_became_peered":"2024-05-06T00:27:54.715397+0000","last_unstale":"2024-05-06T00:27:56.916383+0000","last_undegraded":"2024-05-06T00:27:56.916383+0000","last_fullsized":"2024-05-06T00:27:56.916383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:41:25.546409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":12328583,"num_objects":30,"num_object_clones":4,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":15,"num_read":3511,"num_read_kb":26147,"num_write":235,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":152199,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"75'147","reported_seq":2459,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.904513+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:27:56.904513+0000","last_peered":"2024-05-06T00:27:56.904513+0000","last_clean":"2024-05-06T00:27:56.904513+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:27:56.904513+0000","last_undegraded":"2024-05-06T00:27:56.904513+0000","last_fullsized":"2024-05-06T00:27:56.904513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:23:47.978385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":50371087,"num_objects":39,"num_object_clones":9,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":15,"num_read":2249,"num_read_kb":26381,"num_write":135,"num_write_kb":13616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"74'140","reported_seq":983,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724711+0000","last_change":"2024-05-06T00:27:42.622825+0000","last_active":"2024-05-06T00:27:56.724711+0000","last_peered":"2024-05-06T00:27:56.724711+0000","last_clean":"2024-05-06T00:27:56.724711+0000","last_became_active":"2024-05-06T00:27:42.621392+0000","last_became_peered":"2024-05-06T00:27:42.621392+0000","last_unstale":"2024-05-06T00:27:56.724711+0000","last_undegraded":"2024-05-06T00:27:56.724711+0000","last_fullsized":"2024-05-06T00:27:56.724711+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:40:24.689009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":12908544,"num_objects":33,"num_object_clones":7,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":21,"num_read":634,"num_read_kb":7380,"num_write":118,"num_write_kb":780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":69632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"75'447","reported_seq":5574,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.725093+0000","last_change":"2024-05-06T00:27:42.622974+0000","last_active":"2024-05-06T00:27:56.725093+0000","last_peered":"2024-05-06T00:27:56.725093+0000","last_clean":"2024-05-06T00:27:56.725093+0000","last_became_active":"2024-05-06T00:27:42.621111+0000","last_became_peered":"2024-05-06T00:27:42.621111+0000","last_unstale":"2024-05-06T00:27:56.725093+0000","last_undegraded":"2024-05-06T00:27:56.725093+0000","last_fullsized":"2024-05-06T00:27:56.725093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:30:12.166756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":16783925,"num_objects":37,"num_object_clones":9,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":17,"num_read":7069,"num_read_kb":23099,"num_write":420,"num_write_kb":3347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":510005,"num_keys_recovered":60,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"74'500","reported_seq":5300,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724717+0000","last_change":"2024-05-06T00:27:54.710688+0000","last_active":"2024-05-06T00:27:56.724717+0000","last_peered":"2024-05-06T00:27:56.724717+0000","last_clean":"2024-05-06T00:27:56.724717+0000","last_became_active":"2024-05-06T00:27:54.709962+0000","last_became_peered":"2024-05-06T00:27:54.709962+0000","last_unstale":"2024-05-06T00:27:56.724717+0000","last_undegraded":"2024-05-06T00:27:56.724717+0000","last_fullsized":"2024-05-06T00:27:56.724717+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:55:21.994779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":45521482,"num_objects":60,"num_object_clones":18,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":28,"num_read":6629,"num_read_kb":22609,"num_write":450,"num_write_kb":3631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":306250,"num_keys_recovered":35,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"75'238","reported_seq":3675,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.892970+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:27:56.892970+0000","last_peered":"2024-05-06T00:27:56.892970+0000","last_clean":"2024-05-06T00:27:56.892970+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:27:56.892970+0000","last_undegraded":"2024-05-06T00:27:56.892970+0000","last_fullsized":"2024-05-06T00:27:56.892970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:52:07.176145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":24558249,"num_objects":35,"num_object_clones":5,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":15,"num_read":4151,"num_read_kb":19362,"num_write":281,"num_write_kb":10887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"73'181","reported_seq":1616,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.738947+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:27:56.738947+0000","last_peered":"2024-05-06T00:27:56.738947+0000","last_clean":"2024-05-06T00:27:56.738947+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:27:56.738947+0000","last_undegraded":"2024-05-06T00:27:56.738947+0000","last_fullsized":"2024-05-06T00:27:56.738947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:59:17.528639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":17097812,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":13,"num_read":2037,"num_read_kb":8612,"num_write":148,"num_write_kb":3656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"74'181","reported_seq":2549,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.920160+0000","last_change":"2024-05-06T00:27:54.712678+0000","last_active":"2024-05-06T00:27:56.920160+0000","last_peered":"2024-05-06T00:27:56.920160+0000","last_clean":"2024-05-06T00:27:56.920160+0000","last_became_active":"2024-05-06T00:27:54.711993+0000","last_became_peered":"2024-05-06T00:27:54.711993+0000","last_unstale":"2024-05-06T00:27:56.920160+0000","last_undegraded":"2024-05-06T00:27:56.920160+0000","last_fullsized":"2024-05-06T00:27:56.920160+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:24.745539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":8834064,"num_objects":27,"num_object_clones":5,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":13,"num_read":2379,"num_read_kb":10332,"num_write":139,"num_write_kb":819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":116240,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"73'167","reported_seq":1801,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.716490+0000","last_change":"2024-05-06T00:27:54.717281+0000","last_active":"2024-05-06T00:27:56.716490+0000","last_peered":"2024-05-06T00:27:56.716490+0000","last_clean":"2024-05-06T00:27:56.716490+0000","last_became_active":"2024-05-06T00:27:54.715956+0000","last_became_peered":"2024-05-06T00:27:54.715956+0000","last_unstale":"2024-05-06T00:27:56.716490+0000","last_undegraded":"2024-05-06T00:27:56.716490+0000","last_fullsized":"2024-05-06T00:27:56.716490+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:32:14.879921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":41171498,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":18,"num_read":1385,"num_read_kb":7378,"num_write":149,"num_write_kb":3224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":579156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"71'170","reported_seq":1339,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739328+0000","last_change":"2024-05-06T00:27:42.620191+0000","last_active":"2024-05-06T00:27:56.739328+0000","last_peered":"2024-05-06T00:27:56.739328+0000","last_clean":"2024-05-06T00:27:56.739328+0000","last_became_active":"2024-05-06T00:27:42.619546+0000","last_became_peered":"2024-05-06T00:27:42.619546+0000","last_unstale":"2024-05-06T00:27:56.739328+0000","last_undegraded":"2024-05-06T00:27:56.739328+0000","last_fullsized":"2024-05-06T00:27:56.739328+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:03:56.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":39371331,"num_objects":36,"num_object_clones":10,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":18,"num_read":1321,"num_read_kb":11134,"num_write":191,"num_write_kb":772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":240711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"75'127","reported_seq":1407,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.838048+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:27:56.838048+0000","last_peered":"2024-05-06T00:27:56.838048+0000","last_clean":"2024-05-06T00:27:56.838048+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:27:56.838048+0000","last_undegraded":"2024-05-06T00:27:56.838048+0000","last_fullsized":"2024-05-06T00:27:56.838048+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:37:36.229063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10709504,"num_objects":35,"num_object_clones":9,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":20,"num_read":1210,"num_read_kb":4604,"num_write":111,"num_write_kb":12132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"75'122","reported_seq":1160,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.886126+0000","last_change":"2024-05-06T00:27:21.967099+0000","last_active":"2024-05-06T00:27:56.886126+0000","last_peered":"2024-05-06T00:27:56.886126+0000","last_clean":"2024-05-06T00:27:56.886126+0000","last_became_active":"2024-05-06T00:27:21.964399+0000","last_became_peered":"2024-05-06T00:27:21.964399+0000","last_unstale":"2024-05-06T00:27:56.886126+0000","last_undegraded":"2024-05-06T00:27:56.886126+0000","last_fullsized":"2024-05-06T00:27:56.886126+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:52:48.084587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":18752528,"num_objects":26,"num_object_clones":4,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":13,"num_read":929,"num_read_kb":8469,"num_write":102,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.725092+0000","last_change":"2024-05-06T00:27:21.965066+0000","last_active":"2024-05-06T00:27:56.725092+0000","last_peered":"2024-05-06T00:27:56.725092+0000","last_clean":"2024-05-06T00:27:56.725092+0000","last_became_active":"2024-05-06T00:27:21.964272+0000","last_became_peered":"2024-05-06T00:27:21.964272+0000","last_unstale":"2024-05-06T00:27:56.725092+0000","last_undegraded":"2024-05-06T00:27:56.725092+0000","last_fullsized":"2024-05-06T00:27:56.725092+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:06.226485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"74'706","reported_seq":4904,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739212+0000","last_change":"2024-05-06T00:27:54.712069+0000","last_active":"2024-05-06T00:27:56.739212+0000","last_peered":"2024-05-06T00:27:56.739212+0000","last_clean":"2024-05-06T00:27:56.739212+0000","last_became_active":"2024-05-06T00:27:54.710729+0000","last_became_peered":"2024-05-06T00:27:54.710729+0000","last_unstale":"2024-05-06T00:27:56.739212+0000","last_undegraded":"2024-05-06T00:27:56.739212+0000","last_fullsized":"2024-05-06T00:27:56.739212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:57:20.577448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":8202358,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":12,"num_read":8323,"num_read_kb":14851,"num_write":871,"num_write_kb":2590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1211033,"num_keys_recovered":35,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":128,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739465+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:27:56.739465+0000","last_peered":"2024-05-06T00:27:56.739465+0000","last_clean":"2024-05-06T00:27:56.739465+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:27:56.739465+0000","last_undegraded":"2024-05-06T00:27:56.739465+0000","last_fullsized":"2024-05-06T00:27:56.739465+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"75'380","reported_seq":2799,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.910193+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:27:56.910193+0000","last_peered":"2024-05-06T00:27:56.910193+0000","last_clean":"2024-05-06T00:27:56.910193+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:27:56.910193+0000","last_undegraded":"2024-05-06T00:27:56.910193+0000","last_fullsized":"2024-05-06T00:27:56.910193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:34:24.663083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":12731941,"num_objects":22,"num_object_clones":4,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":11,"num_read":4379,"num_read_kb":7614,"num_write":451,"num_write_kb":1014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724578+0000","last_change":"2024-05-06T00:27:51.613163+0000","last_active":"2024-05-06T00:27:56.724578+0000","last_peered":"2024-05-06T00:27:56.724578+0000","last_clean":"2024-05-06T00:27:56.724578+0000","last_became_active":"2024-05-06T00:27:51.612735+0000","last_became_peered":"2024-05-06T00:27:51.612735+0000","last_unstale":"2024-05-06T00:27:56.724578+0000","last_undegraded":"2024-05-06T00:27:56.724578+0000","last_fullsized":"2024-05-06T00:27:56.724578+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:12:00.610088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"75'347","reported_seq":1909,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739268+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:27:56.739268+0000","last_peered":"2024-05-06T00:27:56.739268+0000","last_clean":"2024-05-06T00:27:56.739268+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:27:56.739268+0000","last_undegraded":"2024-05-06T00:27:56.739268+0000","last_fullsized":"2024-05-06T00:27:56.739268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:54:52.271587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":26763377,"num_objects":41,"num_object_clones":8,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":22,"num_read":2332,"num_read_kb":13053,"num_write":460,"num_write_kb":2173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":133,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739301+0000","last_change":"2024-05-06T00:27:51.622071+0000","last_active":"2024-05-06T00:27:56.739301+0000","last_peered":"2024-05-06T00:27:56.739301+0000","last_clean":"2024-05-06T00:27:56.739301+0000","last_became_active":"2024-05-06T00:27:51.620072+0000","last_became_peered":"2024-05-06T00:27:51.620072+0000","last_unstale":"2024-05-06T00:27:56.739301+0000","last_undegraded":"2024-05-06T00:27:56.739301+0000","last_fullsized":"2024-05-06T00:27:56.739301+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:46:47.691104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"75'212","reported_seq":1453,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.900332+0000","last_change":"2024-05-06T00:27:42.616022+0000","last_active":"2024-05-06T00:27:56.900332+0000","last_peered":"2024-05-06T00:27:56.900332+0000","last_clean":"2024-05-06T00:27:56.900332+0000","last_became_active":"2024-05-06T00:27:42.615215+0000","last_became_peered":"2024-05-06T00:27:42.615215+0000","last_unstale":"2024-05-06T00:27:56.900332+0000","last_undegraded":"2024-05-06T00:27:56.900332+0000","last_fullsized":"2024-05-06T00:27:56.900332+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:37:51.578091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":3346518,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":12,"num_read":1806,"num_read_kb":6410,"num_write":223,"num_write_kb":868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":51232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"73'92","reported_seq":1085,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724992+0000","last_change":"2024-05-06T00:27:51.701463+0000","last_active":"2024-05-06T00:27:56.724992+0000","last_peered":"2024-05-06T00:27:56.724992+0000","last_clean":"2024-05-06T00:27:56.724992+0000","last_became_active":"2024-05-06T00:27:51.613258+0000","last_became_peered":"2024-05-06T00:27:51.613258+0000","last_unstale":"2024-05-06T00:27:56.724992+0000","last_undegraded":"2024-05-06T00:27:56.724992+0000","last_fullsized":"2024-05-06T00:27:56.724992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:25:02.170593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":5500453,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":12,"num_read":784,"num_read_kb":7719,"num_write":80,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":53285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739207+0000","last_change":"2024-05-06T00:27:51.621163+0000","last_active":"2024-05-06T00:27:56.739207+0000","last_peered":"2024-05-06T00:27:56.739207+0000","last_clean":"2024-05-06T00:27:56.739207+0000","last_became_active":"2024-05-06T00:27:51.620331+0000","last_became_peered":"2024-05-06T00:27:51.620331+0000","last_unstale":"2024-05-06T00:27:56.739207+0000","last_undegraded":"2024-05-06T00:27:56.739207+0000","last_fullsized":"2024-05-06T00:27:56.739207+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:32:09.592916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"71'365","reported_seq":3156,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739057+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:27:56.739057+0000","last_peered":"2024-05-06T00:27:56.739057+0000","last_clean":"2024-05-06T00:27:56.739057+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:27:56.739057+0000","last_undegraded":"2024-05-06T00:27:56.739057+0000","last_fullsized":"2024-05-06T00:27:56.739057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:51:49.852959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":10184838,"num_objects":31,"num_object_clones":4,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":14,"num_read":3985,"num_read_kb":15693,"num_write":544,"num_write_kb":2943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":128,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.714964+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:27:56.714964+0000","last_peered":"2024-05-06T00:27:56.714964+0000","last_clean":"2024-05-06T00:27:56.714964+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:27:56.714964+0000","last_undegraded":"2024-05-06T00:27:56.714964+0000","last_fullsized":"2024-05-06T00:27:56.714964+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"75'132","reported_seq":1363,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.826460+0000","last_change":"2024-05-06T00:27:54.715441+0000","last_active":"2024-05-06T00:27:56.826460+0000","last_peered":"2024-05-06T00:27:56.826460+0000","last_clean":"2024-05-06T00:27:56.826460+0000","last_became_active":"2024-05-06T00:27:54.714980+0000","last_became_peered":"2024-05-06T00:27:54.714980+0000","last_unstale":"2024-05-06T00:27:56.826460+0000","last_undegraded":"2024-05-06T00:27:56.826460+0000","last_fullsized":"2024-05-06T00:27:56.826460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:20:08.954285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":29806080,"num_objects":32,"num_object_clones":7,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":13,"num_read":975,"num_read_kb":8275,"num_write":128,"num_write_kb":1361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":452608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":67,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739076+0000","last_change":"2024-05-06T00:27:21.957697+0000","last_active":"2024-05-06T00:27:56.739076+0000","last_peered":"2024-05-06T00:27:56.739076+0000","last_clean":"2024-05-06T00:27:56.739076+0000","last_became_active":"2024-05-06T00:27:21.957293+0000","last_became_peered":"2024-05-06T00:27:21.957293+0000","last_unstale":"2024-05-06T00:27:56.739076+0000","last_undegraded":"2024-05-06T00:27:56.739076+0000","last_fullsized":"2024-05-06T00:27:56.739076+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:36:37.494499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"74'245","reported_seq":2802,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.885704+0000","last_change":"2024-05-06T00:27:42.620136+0000","last_active":"2024-05-06T00:27:56.885704+0000","last_peered":"2024-05-06T00:27:56.885704+0000","last_clean":"2024-05-06T00:27:56.885704+0000","last_became_active":"2024-05-06T00:27:42.619561+0000","last_became_peered":"2024-05-06T00:27:42.619561+0000","last_unstale":"2024-05-06T00:27:56.885704+0000","last_undegraded":"2024-05-06T00:27:56.885704+0000","last_fullsized":"2024-05-06T00:27:56.885704+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:11:00.816246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":12466250,"num_objects":25,"num_object_clones":5,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":13,"num_read":3551,"num_read_kb":11536,"num_write":179,"num_write_kb":1137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":26698,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.716275+0000","last_change":"2024-05-06T00:27:40.124309+0000","last_active":"2024-05-06T00:27:56.716275+0000","last_peered":"2024-05-06T00:27:56.716275+0000","last_clean":"2024-05-06T00:27:56.716275+0000","last_became_active":"2024-05-06T00:27:40.123433+0000","last_became_peered":"2024-05-06T00:27:40.123433+0000","last_unstale":"2024-05-06T00:27:56.716275+0000","last_undegraded":"2024-05-06T00:27:56.716275+0000","last_fullsized":"2024-05-06T00:27:56.716275+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:54:51.350069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":26,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.738875+0000","last_change":"2024-05-06T00:27:51.714759+0000","last_active":"2024-05-06T00:27:56.738875+0000","last_peered":"2024-05-06T00:27:56.738875+0000","last_clean":"2024-05-06T00:27:56.738875+0000","last_became_active":"2024-05-06T00:27:51.616671+0000","last_became_peered":"2024-05-06T00:27:51.616671+0000","last_unstale":"2024-05-06T00:27:56.738875+0000","last_undegraded":"2024-05-06T00:27:56.738875+0000","last_fullsized":"2024-05-06T00:27:56.738875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:02:48.775028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"74'326","reported_seq":1879,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.715011+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:27:56.715011+0000","last_peered":"2024-05-06T00:27:56.715011+0000","last_clean":"2024-05-06T00:27:56.715011+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:27:56.715011+0000","last_undegraded":"2024-05-06T00:27:56.715011+0000","last_fullsized":"2024-05-06T00:27:56.715011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:40.942592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":23172264,"num_objects":38,"num_object_clones":7,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":20,"num_read":2486,"num_read_kb":4988,"num_write":408,"num_write_kb":1029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"74'338","reported_seq":3681,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724506+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:27:56.724506+0000","last_peered":"2024-05-06T00:27:56.724506+0000","last_clean":"2024-05-06T00:27:56.724506+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:27:56.724506+0000","last_undegraded":"2024-05-06T00:27:56.724506+0000","last_fullsized":"2024-05-06T00:27:56.724506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:15:42.033835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":11876078,"num_objects":37,"num_object_clones":7,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":21,"num_read":4267,"num_read_kb":9330,"num_write":483,"num_write_kb":8579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"74'337","reported_seq":4383,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724508+0000","last_change":"2024-05-06T00:27:21.968588+0000","last_active":"2024-05-06T00:27:56.724508+0000","last_peered":"2024-05-06T00:27:56.724508+0000","last_clean":"2024-05-06T00:27:56.724508+0000","last_became_active":"2024-05-06T00:27:21.967179+0000","last_became_peered":"2024-05-06T00:27:21.967179+0000","last_unstale":"2024-05-06T00:27:56.724508+0000","last_undegraded":"2024-05-06T00:27:56.724508+0000","last_fullsized":"2024-05-06T00:27:56.724508+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:49:21.980303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":23368454,"num_objects":47,"num_object_clones":11,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":22,"num_read":5478,"num_read_kb":24308,"num_write":239,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"75'109","reported_seq":1059,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.912659+0000","last_change":"2024-05-06T00:27:54.710748+0000","last_active":"2024-05-06T00:27:56.912659+0000","last_peered":"2024-05-06T00:27:56.912659+0000","last_clean":"2024-05-06T00:27:56.912659+0000","last_became_active":"2024-05-06T00:27:54.710063+0000","last_became_peered":"2024-05-06T00:27:54.710063+0000","last_unstale":"2024-05-06T00:27:56.912659+0000","last_undegraded":"2024-05-06T00:27:56.912659+0000","last_fullsized":"2024-05-06T00:27:56.912659+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:39:38.634935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":13536424,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":14,"num_read":786,"num_read_kb":12418,"num_write":92,"num_write_kb":4649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":1070760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"71'215","reported_seq":2012,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739148+0000","last_change":"2024-05-06T00:27:54.711021+0000","last_active":"2024-05-06T00:27:56.739148+0000","last_peered":"2024-05-06T00:27:56.739148+0000","last_clean":"2024-05-06T00:27:56.739148+0000","last_became_active":"2024-05-06T00:27:54.709611+0000","last_became_peered":"2024-05-06T00:27:54.709611+0000","last_unstale":"2024-05-06T00:27:56.739148+0000","last_undegraded":"2024-05-06T00:27:56.739148+0000","last_fullsized":"2024-05-06T00:27:56.739148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:56:55.118050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":5913637,"num_objects":31,"num_object_clones":5,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":19,"num_read":1857,"num_read_kb":7012,"num_write":175,"num_write_kb":1139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":139301,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"75'578","reported_seq":3559,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.716566+0000","last_change":"2024-05-06T00:27:21.957565+0000","last_active":"2024-05-06T00:27:56.716566+0000","last_peered":"2024-05-06T00:27:56.716566+0000","last_clean":"2024-05-06T00:27:56.716566+0000","last_became_active":"2024-05-06T00:27:21.955822+0000","last_became_peered":"2024-05-06T00:27:21.955822+0000","last_unstale":"2024-05-06T00:27:56.716566+0000","last_undegraded":"2024-05-06T00:27:56.716566+0000","last_fullsized":"2024-05-06T00:27:56.716566+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:15:05.768048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":11875913,"num_objects":39,"num_object_clones":7,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":23,"num_read":5209,"num_read_kb":13535,"num_write":921,"num_write_kb":1516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"75'314","reported_seq":1946,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.885614+0000","last_change":"2024-05-06T00:27:42.622702+0000","last_active":"2024-05-06T00:27:56.885614+0000","last_peered":"2024-05-06T00:27:56.885614+0000","last_clean":"2024-05-06T00:27:56.885614+0000","last_became_active":"2024-05-06T00:27:42.621212+0000","last_became_peered":"2024-05-06T00:27:42.621212+0000","last_unstale":"2024-05-06T00:27:56.885614+0000","last_undegraded":"2024-05-06T00:27:56.885614+0000","last_fullsized":"2024-05-06T00:27:56.885614+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:47:20.958627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":17996926,"num_objects":33,"num_object_clones":5,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":17,"num_read":2937,"num_read_kb":7767,"num_write":420,"num_write_kb":2473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1517694,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"73'143","reported_seq":868,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739536+0000","last_change":"2024-05-06T00:27:54.711546+0000","last_active":"2024-05-06T00:27:56.739536+0000","last_peered":"2024-05-06T00:27:56.739536+0000","last_clean":"2024-05-06T00:27:56.739536+0000","last_became_active":"2024-05-06T00:27:54.710473+0000","last_became_peered":"2024-05-06T00:27:54.710473+0000","last_unstale":"2024-05-06T00:27:56.739536+0000","last_undegraded":"2024-05-06T00:27:56.739536+0000","last_fullsized":"2024-05-06T00:27:56.739536+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:18:08.964424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":5859156,"num_objects":28,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":15,"num_read":842,"num_read_kb":6946,"num_write":160,"num_write_kb":585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":120488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"75'283","reported_seq":4489,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.850486+0000","last_change":"2024-05-06T00:27:54.717002+0000","last_active":"2024-05-06T00:27:56.850486+0000","last_peered":"2024-05-06T00:27:56.850486+0000","last_clean":"2024-05-06T00:27:56.850486+0000","last_became_active":"2024-05-06T00:27:54.715616+0000","last_became_peered":"2024-05-06T00:27:54.715616+0000","last_unstale":"2024-05-06T00:27:56.850486+0000","last_undegraded":"2024-05-06T00:27:56.850486+0000","last_fullsized":"2024-05-06T00:27:56.850486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:10:43.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":25476096,"num_objects":31,"num_object_clones":4,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":15,"num_read":5583,"num_read_kb":11820,"num_write":188,"num_write_kb":2817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":432128,"num_keys_recovered":34,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"75'150","reported_seq":1103,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.830983+0000","last_change":"2024-05-06T00:27:21.968173+0000","last_active":"2024-05-06T00:27:56.830983+0000","last_peered":"2024-05-06T00:27:56.830983+0000","last_clean":"2024-05-06T00:27:56.830983+0000","last_became_active":"2024-05-06T00:27:21.965833+0000","last_became_peered":"2024-05-06T00:27:21.965833+0000","last_unstale":"2024-05-06T00:27:56.830983+0000","last_undegraded":"2024-05-06T00:27:56.830983+0000","last_fullsized":"2024-05-06T00:27:56.830983+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:09:52.355002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":22814764,"num_objects":33,"num_object_clones":3,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":20,"num_read":970,"num_read_kb":5949,"num_write":178,"num_write_kb":1430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"75'193","reported_seq":3914,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.881934+0000","last_change":"2024-05-06T00:27:54.711712+0000","last_active":"2024-05-06T00:27:56.881934+0000","last_peered":"2024-05-06T00:27:56.881934+0000","last_clean":"2024-05-06T00:27:56.881934+0000","last_became_active":"2024-05-06T00:27:54.710459+0000","last_became_peered":"2024-05-06T00:27:54.710459+0000","last_unstale":"2024-05-06T00:27:56.881934+0000","last_undegraded":"2024-05-06T00:27:56.881934+0000","last_fullsized":"2024-05-06T00:27:56.881934+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:09:03.012622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":31332864,"num_objects":46,"num_object_clones":12,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":23,"num_read":3624,"num_read_kb":13631,"num_write":191,"num_write_kb":2046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":477184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"73'281","reported_seq":6170,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.714903+0000","last_change":"2024-05-06T00:27:42.615243+0000","last_active":"2024-05-06T00:27:56.714903+0000","last_peered":"2024-05-06T00:27:56.714903+0000","last_clean":"2024-05-06T00:27:56.714903+0000","last_became_active":"2024-05-06T00:27:42.614475+0000","last_became_peered":"2024-05-06T00:27:42.614475+0000","last_unstale":"2024-05-06T00:27:56.714903+0000","last_undegraded":"2024-05-06T00:27:56.714903+0000","last_fullsized":"2024-05-06T00:27:56.714903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:01:55.371583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":26105441,"num_objects":35,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":14,"num_read":6682,"num_read_kb":22988,"num_write":209,"num_write_kb":3244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2163809,"num_keys_recovered":23,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":626738439,"num_objects":1070,"num_object_clones":214,"num_object_copies":2140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":538,"num_read":100360,"num_read_kb":401350,"num_write":9079,"num_write_kb":97931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1108,"num_bytes_recovered":9690413,"num_keys_recovered":241,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":60448768,"data_stored":89529075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8366,"ondisk_log_size":8366,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640277,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57852,"kb_used_data":14192,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94313988,"statfs":{"total":96636764160,"available":96577523712,"internally_reserved":0,"allocated":14532608,"data_stored":22786179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640277,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50608,"kb_used_data":11112,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94321232,"statfs":{"total":96636764160,"available":96584941568,"internally_reserved":0,"allocated":11378688,"data_stored":14754905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51572,"kb_used_data":12076,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94320268,"statfs":{"total":96636764160,"available":96583954432,"internally_reserved":0,"allocated":12365824,"data_stored":15500654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640279,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59904,"kb_used_data":16244,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94311936,"statfs":{"total":96636764160,"available":96575422464,"internally_reserved":0,"allocated":16633856,"data_stored":25022388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":47244640277,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4857856,"data_stored":6773154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15167488,"data_stored":24079016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11493376,"data_stored":15147650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9912320,"data_stored":13811533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13660160,"data_stored":22433175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5357568,"data_stored":7284547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:28:00.144 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:28:00.188 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:00.190+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:00.211 INFO:teuthology.orchestra.run.smithi186.stdout:2986 clone 23 order 22 su 131072 sc 12 2024-05-06T00:28:00.268 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:00.267+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:00.288 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:00.288 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:00.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:00.389 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:00.387+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:28:00.455 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:28:00.467 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 90.862504697000006, 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:28:00.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:28:00.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589855997999997, 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:28:00.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:28:00.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:28:00.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:00.481 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589820822, 2024-05-06T00:28:00.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:28:00.481 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.481 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.481 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:28:00.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:28:00.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589732142999999, 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.487 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:28:00.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589630200000002, 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:00.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:28:00.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:28:00.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589521097999999, 2024-05-06T00:28:00.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:28:00.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:28:00.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589423238999998, 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.506 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:28:00.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:28:00.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:28:00.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:28:00.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:28:00.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589358189000002, 2024-05-06T00:28:00.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589265543000003, 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.518 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:28:00.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.589193539, 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.524 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:28:00.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:28:00.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829732053000001, 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.530 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:28:00.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829712805, 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.536 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:28:00.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829662685000002, 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.542 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:28:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:28:00.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829641633999998, 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.548 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:28:00.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:28:00.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829626314999999, 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.559 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829609433000002, 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:28:00.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.565 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829591096999998, 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:00.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:28:00.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.571 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829575095000003, 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:28:00.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.829488732000002, 2024-05-06T00:28:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:28:00.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.819526256000003, 2024-05-06T00:28:00.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:28:00.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:28:00.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:00.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:00.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:00.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:28:00.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:28:00.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.589 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:00.589 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:00.589 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:00.591 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-05-06T00:28:00.416047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":627328807,"num_objects":1072,"num_object_clones":214,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1072,"num_whiteouts":538,"num_read":100426,"num_read_kb":401405,"num_write":9180,"num_write_kb":99559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":10280781,"num_keys_recovered":241,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8438,"ondisk_log_size":8438,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":219936,"kb_used_data":53624,"kb_used_omap":6,"kb_used_meta":166265,"kb_avail":377267424,"statfs":{"total":386547056640,"available":386321842176,"internally_reserved":0,"allocated":54910976,"data_stored":78064126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":170256150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41098404,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":6100,"num_read_kb":14703,"num_write":666,"num_write_kb":3389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":4464034,"num_keys_recovered":103,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001042"},"pg_stats":[{"pgid":"3.1f","version":"74'200","reported_seq":3602,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.916383+0000","last_change":"2024-05-06T00:27:54.717030+0000","last_active":"2024-05-06T00:27:56.916383+0000","last_peered":"2024-05-06T00:27:56.916383+0000","last_clean":"2024-05-06T00:27:56.916383+0000","last_became_active":"2024-05-06T00:27:54.715397+0000","last_became_peered":"2024-05-06T00:27:54.715397+0000","last_unstale":"2024-05-06T00:27:56.916383+0000","last_undegraded":"2024-05-06T00:27:56.916383+0000","last_fullsized":"2024-05-06T00:27:56.916383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:41:25.546409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":12328583,"num_objects":30,"num_object_clones":4,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":15,"num_read":3511,"num_read_kb":26147,"num_write":235,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":152199,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"75'147","reported_seq":2459,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.904513+0000","last_change":"2024-05-06T00:26:45.715149+0000","last_active":"2024-05-06T00:27:56.904513+0000","last_peered":"2024-05-06T00:27:56.904513+0000","last_clean":"2024-05-06T00:27:56.904513+0000","last_became_active":"2024-05-06T00:26:43.690861+0000","last_became_peered":"2024-05-06T00:26:43.690861+0000","last_unstale":"2024-05-06T00:27:56.904513+0000","last_undegraded":"2024-05-06T00:27:56.904513+0000","last_fullsized":"2024-05-06T00:27:56.904513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:23:47.978385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":50371087,"num_objects":39,"num_object_clones":9,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":15,"num_read":2249,"num_read_kb":26381,"num_write":135,"num_write_kb":13616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"74'140","reported_seq":983,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724711+0000","last_change":"2024-05-06T00:27:42.622825+0000","last_active":"2024-05-06T00:27:56.724711+0000","last_peered":"2024-05-06T00:27:56.724711+0000","last_clean":"2024-05-06T00:27:56.724711+0000","last_became_active":"2024-05-06T00:27:42.621392+0000","last_became_peered":"2024-05-06T00:27:42.621392+0000","last_unstale":"2024-05-06T00:27:56.724711+0000","last_undegraded":"2024-05-06T00:27:56.724711+0000","last_fullsized":"2024-05-06T00:27:56.724711+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:40:24.689009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":12908544,"num_objects":33,"num_object_clones":7,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":21,"num_read":634,"num_read_kb":7380,"num_write":118,"num_write_kb":780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":69632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"75'447","reported_seq":5574,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.725093+0000","last_change":"2024-05-06T00:27:42.622974+0000","last_active":"2024-05-06T00:27:56.725093+0000","last_peered":"2024-05-06T00:27:56.725093+0000","last_clean":"2024-05-06T00:27:56.725093+0000","last_became_active":"2024-05-06T00:27:42.621111+0000","last_became_peered":"2024-05-06T00:27:42.621111+0000","last_unstale":"2024-05-06T00:27:56.725093+0000","last_undegraded":"2024-05-06T00:27:56.725093+0000","last_fullsized":"2024-05-06T00:27:56.725093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:30:12.166756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":16783925,"num_objects":37,"num_object_clones":9,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":17,"num_read":7069,"num_read_kb":23099,"num_write":420,"num_write_kb":3347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":510005,"num_keys_recovered":60,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"74'500","reported_seq":5300,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724717+0000","last_change":"2024-05-06T00:27:54.710688+0000","last_active":"2024-05-06T00:27:56.724717+0000","last_peered":"2024-05-06T00:27:56.724717+0000","last_clean":"2024-05-06T00:27:56.724717+0000","last_became_active":"2024-05-06T00:27:54.709962+0000","last_became_peered":"2024-05-06T00:27:54.709962+0000","last_unstale":"2024-05-06T00:27:56.724717+0000","last_undegraded":"2024-05-06T00:27:56.724717+0000","last_fullsized":"2024-05-06T00:27:56.724717+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:55:21.994779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":45521482,"num_objects":60,"num_object_clones":18,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":28,"num_read":6629,"num_read_kb":22609,"num_write":450,"num_write_kb":3631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":306250,"num_keys_recovered":35,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"75'238","reported_seq":3675,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.892970+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:27:56.892970+0000","last_peered":"2024-05-06T00:27:56.892970+0000","last_clean":"2024-05-06T00:27:56.892970+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:27:56.892970+0000","last_undegraded":"2024-05-06T00:27:56.892970+0000","last_fullsized":"2024-05-06T00:27:56.892970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:52:07.176145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":24558249,"num_objects":35,"num_object_clones":5,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":15,"num_read":4151,"num_read_kb":19362,"num_write":281,"num_write_kb":10887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"73'181","reported_seq":1616,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.738947+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:27:56.738947+0000","last_peered":"2024-05-06T00:27:56.738947+0000","last_clean":"2024-05-06T00:27:56.738947+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:27:56.738947+0000","last_undegraded":"2024-05-06T00:27:56.738947+0000","last_fullsized":"2024-05-06T00:27:56.738947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:59:17.528639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":17097812,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":13,"num_read":2037,"num_read_kb":8612,"num_write":148,"num_write_kb":3656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"74'181","reported_seq":2549,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.920160+0000","last_change":"2024-05-06T00:27:54.712678+0000","last_active":"2024-05-06T00:27:56.920160+0000","last_peered":"2024-05-06T00:27:56.920160+0000","last_clean":"2024-05-06T00:27:56.920160+0000","last_became_active":"2024-05-06T00:27:54.711993+0000","last_became_peered":"2024-05-06T00:27:54.711993+0000","last_unstale":"2024-05-06T00:27:56.920160+0000","last_undegraded":"2024-05-06T00:27:56.920160+0000","last_fullsized":"2024-05-06T00:27:56.920160+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:24.745539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":8834064,"num_objects":27,"num_object_clones":5,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":13,"num_read":2379,"num_read_kb":10332,"num_write":139,"num_write_kb":819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":116240,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"73'167","reported_seq":1801,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.716490+0000","last_change":"2024-05-06T00:27:54.717281+0000","last_active":"2024-05-06T00:27:56.716490+0000","last_peered":"2024-05-06T00:27:56.716490+0000","last_clean":"2024-05-06T00:27:56.716490+0000","last_became_active":"2024-05-06T00:27:54.715956+0000","last_became_peered":"2024-05-06T00:27:54.715956+0000","last_unstale":"2024-05-06T00:27:56.716490+0000","last_undegraded":"2024-05-06T00:27:56.716490+0000","last_fullsized":"2024-05-06T00:27:56.716490+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:32:14.879921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":41171498,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":18,"num_read":1385,"num_read_kb":7378,"num_write":149,"num_write_kb":3224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":579156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"71'170","reported_seq":1339,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739328+0000","last_change":"2024-05-06T00:27:42.620191+0000","last_active":"2024-05-06T00:27:56.739328+0000","last_peered":"2024-05-06T00:27:56.739328+0000","last_clean":"2024-05-06T00:27:56.739328+0000","last_became_active":"2024-05-06T00:27:42.619546+0000","last_became_peered":"2024-05-06T00:27:42.619546+0000","last_unstale":"2024-05-06T00:27:56.739328+0000","last_undegraded":"2024-05-06T00:27:56.739328+0000","last_fullsized":"2024-05-06T00:27:56.739328+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:03:56.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":39371331,"num_objects":36,"num_object_clones":10,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":18,"num_read":1321,"num_read_kb":11134,"num_write":191,"num_write_kb":772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":240711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"75'127","reported_seq":1407,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.838048+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:27:56.838048+0000","last_peered":"2024-05-06T00:27:56.838048+0000","last_clean":"2024-05-06T00:27:56.838048+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:27:56.838048+0000","last_undegraded":"2024-05-06T00:27:56.838048+0000","last_fullsized":"2024-05-06T00:27:56.838048+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:37:36.229063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10709504,"num_objects":35,"num_object_clones":9,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":20,"num_read":1210,"num_read_kb":4604,"num_write":111,"num_write_kb":12132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"75'122","reported_seq":1160,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.886126+0000","last_change":"2024-05-06T00:27:21.967099+0000","last_active":"2024-05-06T00:27:56.886126+0000","last_peered":"2024-05-06T00:27:56.886126+0000","last_clean":"2024-05-06T00:27:56.886126+0000","last_became_active":"2024-05-06T00:27:21.964399+0000","last_became_peered":"2024-05-06T00:27:21.964399+0000","last_unstale":"2024-05-06T00:27:56.886126+0000","last_undegraded":"2024-05-06T00:27:56.886126+0000","last_fullsized":"2024-05-06T00:27:56.886126+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:52:48.084587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":18752528,"num_objects":26,"num_object_clones":4,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":13,"num_read":929,"num_read_kb":8469,"num_write":102,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.725092+0000","last_change":"2024-05-06T00:27:21.965066+0000","last_active":"2024-05-06T00:27:56.725092+0000","last_peered":"2024-05-06T00:27:56.725092+0000","last_clean":"2024-05-06T00:27:56.725092+0000","last_became_active":"2024-05-06T00:27:21.964272+0000","last_became_peered":"2024-05-06T00:27:21.964272+0000","last_unstale":"2024-05-06T00:27:56.725092+0000","last_undegraded":"2024-05-06T00:27:56.725092+0000","last_fullsized":"2024-05-06T00:27:56.725092+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:06.226485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"74'706","reported_seq":4904,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739212+0000","last_change":"2024-05-06T00:27:54.712069+0000","last_active":"2024-05-06T00:27:56.739212+0000","last_peered":"2024-05-06T00:27:56.739212+0000","last_clean":"2024-05-06T00:27:56.739212+0000","last_became_active":"2024-05-06T00:27:54.710729+0000","last_became_peered":"2024-05-06T00:27:54.710729+0000","last_unstale":"2024-05-06T00:27:56.739212+0000","last_undegraded":"2024-05-06T00:27:56.739212+0000","last_fullsized":"2024-05-06T00:27:56.739212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:57:20.577448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":8202358,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":12,"num_read":8323,"num_read_kb":14851,"num_write":871,"num_write_kb":2590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1211033,"num_keys_recovered":35,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":128,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739465+0000","last_change":"2024-05-06T00:26:35.646844+0000","last_active":"2024-05-06T00:27:56.739465+0000","last_peered":"2024-05-06T00:27:56.739465+0000","last_clean":"2024-05-06T00:27:56.739465+0000","last_became_active":"2024-05-06T00:26:35.646300+0000","last_became_peered":"2024-05-06T00:26:35.646300+0000","last_unstale":"2024-05-06T00:27:56.739465+0000","last_undegraded":"2024-05-06T00:27:56.739465+0000","last_fullsized":"2024-05-06T00:27:56.739465+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:07.606650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"75'380","reported_seq":2799,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.910193+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:27:56.910193+0000","last_peered":"2024-05-06T00:27:56.910193+0000","last_clean":"2024-05-06T00:27:56.910193+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:27:56.910193+0000","last_undegraded":"2024-05-06T00:27:56.910193+0000","last_fullsized":"2024-05-06T00:27:56.910193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:34:24.663083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":12731941,"num_objects":22,"num_object_clones":4,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":11,"num_read":4379,"num_read_kb":7614,"num_write":451,"num_write_kb":1014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724578+0000","last_change":"2024-05-06T00:27:51.613163+0000","last_active":"2024-05-06T00:27:56.724578+0000","last_peered":"2024-05-06T00:27:56.724578+0000","last_clean":"2024-05-06T00:27:56.724578+0000","last_became_active":"2024-05-06T00:27:51.612735+0000","last_became_peered":"2024-05-06T00:27:51.612735+0000","last_unstale":"2024-05-06T00:27:56.724578+0000","last_undegraded":"2024-05-06T00:27:56.724578+0000","last_fullsized":"2024-05-06T00:27:56.724578+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:12:00.610088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"75'347","reported_seq":1909,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739268+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:27:56.739268+0000","last_peered":"2024-05-06T00:27:56.739268+0000","last_clean":"2024-05-06T00:27:56.739268+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:27:56.739268+0000","last_undegraded":"2024-05-06T00:27:56.739268+0000","last_fullsized":"2024-05-06T00:27:56.739268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:54:52.271587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":26763377,"num_objects":41,"num_object_clones":8,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":22,"num_read":2332,"num_read_kb":13053,"num_write":460,"num_write_kb":2173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":133,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739301+0000","last_change":"2024-05-06T00:27:51.622071+0000","last_active":"2024-05-06T00:27:56.739301+0000","last_peered":"2024-05-06T00:27:56.739301+0000","last_clean":"2024-05-06T00:27:56.739301+0000","last_became_active":"2024-05-06T00:27:51.620072+0000","last_became_peered":"2024-05-06T00:27:51.620072+0000","last_unstale":"2024-05-06T00:27:56.739301+0000","last_undegraded":"2024-05-06T00:27:56.739301+0000","last_fullsized":"2024-05-06T00:27:56.739301+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:46:47.691104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"75'212","reported_seq":1453,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.900332+0000","last_change":"2024-05-06T00:27:42.616022+0000","last_active":"2024-05-06T00:27:56.900332+0000","last_peered":"2024-05-06T00:27:56.900332+0000","last_clean":"2024-05-06T00:27:56.900332+0000","last_became_active":"2024-05-06T00:27:42.615215+0000","last_became_peered":"2024-05-06T00:27:42.615215+0000","last_unstale":"2024-05-06T00:27:56.900332+0000","last_undegraded":"2024-05-06T00:27:56.900332+0000","last_fullsized":"2024-05-06T00:27:56.900332+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:37:51.578091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":3346518,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":12,"num_read":1806,"num_read_kb":6410,"num_write":223,"num_write_kb":868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":51232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"73'92","reported_seq":1085,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724992+0000","last_change":"2024-05-06T00:27:51.701463+0000","last_active":"2024-05-06T00:27:56.724992+0000","last_peered":"2024-05-06T00:27:56.724992+0000","last_clean":"2024-05-06T00:27:56.724992+0000","last_became_active":"2024-05-06T00:27:51.613258+0000","last_became_peered":"2024-05-06T00:27:51.613258+0000","last_unstale":"2024-05-06T00:27:56.724992+0000","last_undegraded":"2024-05-06T00:27:56.724992+0000","last_fullsized":"2024-05-06T00:27:56.724992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:25:02.170593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":5500453,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":12,"num_read":784,"num_read_kb":7719,"num_write":80,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":53285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739207+0000","last_change":"2024-05-06T00:27:51.621163+0000","last_active":"2024-05-06T00:27:56.739207+0000","last_peered":"2024-05-06T00:27:56.739207+0000","last_clean":"2024-05-06T00:27:56.739207+0000","last_became_active":"2024-05-06T00:27:51.620331+0000","last_became_peered":"2024-05-06T00:27:51.620331+0000","last_unstale":"2024-05-06T00:27:56.739207+0000","last_undegraded":"2024-05-06T00:27:56.739207+0000","last_fullsized":"2024-05-06T00:27:56.739207+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:32:09.592916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"71'365","reported_seq":3156,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739057+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:27:56.739057+0000","last_peered":"2024-05-06T00:27:56.739057+0000","last_clean":"2024-05-06T00:27:56.739057+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:27:56.739057+0000","last_undegraded":"2024-05-06T00:27:56.739057+0000","last_fullsized":"2024-05-06T00:27:56.739057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:51:49.852959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":10184838,"num_objects":31,"num_object_clones":4,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":14,"num_read":3985,"num_read_kb":15693,"num_write":544,"num_write_kb":2943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":128,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.714964+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:27:56.714964+0000","last_peered":"2024-05-06T00:27:56.714964+0000","last_clean":"2024-05-06T00:27:56.714964+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:27:56.714964+0000","last_undegraded":"2024-05-06T00:27:56.714964+0000","last_fullsized":"2024-05-06T00:27:56.714964+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"75'132","reported_seq":1363,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.826460+0000","last_change":"2024-05-06T00:27:54.715441+0000","last_active":"2024-05-06T00:27:56.826460+0000","last_peered":"2024-05-06T00:27:56.826460+0000","last_clean":"2024-05-06T00:27:56.826460+0000","last_became_active":"2024-05-06T00:27:54.714980+0000","last_became_peered":"2024-05-06T00:27:54.714980+0000","last_unstale":"2024-05-06T00:27:56.826460+0000","last_undegraded":"2024-05-06T00:27:56.826460+0000","last_fullsized":"2024-05-06T00:27:56.826460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:20:08.954285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":29806080,"num_objects":32,"num_object_clones":7,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":13,"num_read":975,"num_read_kb":8275,"num_write":128,"num_write_kb":1361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":452608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":67,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739076+0000","last_change":"2024-05-06T00:27:21.957697+0000","last_active":"2024-05-06T00:27:56.739076+0000","last_peered":"2024-05-06T00:27:56.739076+0000","last_clean":"2024-05-06T00:27:56.739076+0000","last_became_active":"2024-05-06T00:27:21.957293+0000","last_became_peered":"2024-05-06T00:27:21.957293+0000","last_unstale":"2024-05-06T00:27:56.739076+0000","last_undegraded":"2024-05-06T00:27:56.739076+0000","last_fullsized":"2024-05-06T00:27:56.739076+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:36:37.494499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"74'245","reported_seq":2802,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.885704+0000","last_change":"2024-05-06T00:27:42.620136+0000","last_active":"2024-05-06T00:27:56.885704+0000","last_peered":"2024-05-06T00:27:56.885704+0000","last_clean":"2024-05-06T00:27:56.885704+0000","last_became_active":"2024-05-06T00:27:42.619561+0000","last_became_peered":"2024-05-06T00:27:42.619561+0000","last_unstale":"2024-05-06T00:27:56.885704+0000","last_undegraded":"2024-05-06T00:27:56.885704+0000","last_fullsized":"2024-05-06T00:27:56.885704+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:11:00.816246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":12466250,"num_objects":25,"num_object_clones":5,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":13,"num_read":3551,"num_read_kb":11536,"num_write":179,"num_write_kb":1137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":26698,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.716275+0000","last_change":"2024-05-06T00:27:40.124309+0000","last_active":"2024-05-06T00:27:56.716275+0000","last_peered":"2024-05-06T00:27:56.716275+0000","last_clean":"2024-05-06T00:27:56.716275+0000","last_became_active":"2024-05-06T00:27:40.123433+0000","last_became_peered":"2024-05-06T00:27:40.123433+0000","last_unstale":"2024-05-06T00:27:56.716275+0000","last_undegraded":"2024-05-06T00:27:56.716275+0000","last_fullsized":"2024-05-06T00:27:56.716275+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:54:51.350069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":26,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.738875+0000","last_change":"2024-05-06T00:27:51.714759+0000","last_active":"2024-05-06T00:27:56.738875+0000","last_peered":"2024-05-06T00:27:56.738875+0000","last_clean":"2024-05-06T00:27:56.738875+0000","last_became_active":"2024-05-06T00:27:51.616671+0000","last_became_peered":"2024-05-06T00:27:51.616671+0000","last_unstale":"2024-05-06T00:27:56.738875+0000","last_undegraded":"2024-05-06T00:27:56.738875+0000","last_fullsized":"2024-05-06T00:27:56.738875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:02:48.775028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"74'326","reported_seq":1879,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.715011+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:27:56.715011+0000","last_peered":"2024-05-06T00:27:56.715011+0000","last_clean":"2024-05-06T00:27:56.715011+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:27:56.715011+0000","last_undegraded":"2024-05-06T00:27:56.715011+0000","last_fullsized":"2024-05-06T00:27:56.715011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:32:40.942592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":23172264,"num_objects":38,"num_object_clones":7,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":20,"num_read":2486,"num_read_kb":4988,"num_write":408,"num_write_kb":1029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"74'338","reported_seq":3681,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724506+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:27:56.724506+0000","last_peered":"2024-05-06T00:27:56.724506+0000","last_clean":"2024-05-06T00:27:56.724506+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:27:56.724506+0000","last_undegraded":"2024-05-06T00:27:56.724506+0000","last_fullsized":"2024-05-06T00:27:56.724506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:15:42.033835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":11876078,"num_objects":37,"num_object_clones":7,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":21,"num_read":4267,"num_read_kb":9330,"num_write":483,"num_write_kb":8579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"74'337","reported_seq":4383,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.724508+0000","last_change":"2024-05-06T00:27:21.968588+0000","last_active":"2024-05-06T00:27:56.724508+0000","last_peered":"2024-05-06T00:27:56.724508+0000","last_clean":"2024-05-06T00:27:56.724508+0000","last_became_active":"2024-05-06T00:27:21.967179+0000","last_became_peered":"2024-05-06T00:27:21.967179+0000","last_unstale":"2024-05-06T00:27:56.724508+0000","last_undegraded":"2024-05-06T00:27:56.724508+0000","last_fullsized":"2024-05-06T00:27:56.724508+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:49:21.980303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":23368454,"num_objects":47,"num_object_clones":11,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":22,"num_read":5478,"num_read_kb":24308,"num_write":239,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"75'109","reported_seq":1059,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.912659+0000","last_change":"2024-05-06T00:27:54.710748+0000","last_active":"2024-05-06T00:27:56.912659+0000","last_peered":"2024-05-06T00:27:56.912659+0000","last_clean":"2024-05-06T00:27:56.912659+0000","last_became_active":"2024-05-06T00:27:54.710063+0000","last_became_peered":"2024-05-06T00:27:54.710063+0000","last_unstale":"2024-05-06T00:27:56.912659+0000","last_undegraded":"2024-05-06T00:27:56.912659+0000","last_fullsized":"2024-05-06T00:27:56.912659+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:39:38.634935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":13536424,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":14,"num_read":786,"num_read_kb":12418,"num_write":92,"num_write_kb":4649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":1070760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"71'215","reported_seq":2012,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739148+0000","last_change":"2024-05-06T00:27:54.711021+0000","last_active":"2024-05-06T00:27:56.739148+0000","last_peered":"2024-05-06T00:27:56.739148+0000","last_clean":"2024-05-06T00:27:56.739148+0000","last_became_active":"2024-05-06T00:27:54.709611+0000","last_became_peered":"2024-05-06T00:27:54.709611+0000","last_unstale":"2024-05-06T00:27:56.739148+0000","last_undegraded":"2024-05-06T00:27:56.739148+0000","last_fullsized":"2024-05-06T00:27:56.739148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:56:55.118050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":5913637,"num_objects":31,"num_object_clones":5,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":19,"num_read":1857,"num_read_kb":7012,"num_write":175,"num_write_kb":1139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":139301,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"75'578","reported_seq":3559,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.716566+0000","last_change":"2024-05-06T00:27:21.957565+0000","last_active":"2024-05-06T00:27:56.716566+0000","last_peered":"2024-05-06T00:27:56.716566+0000","last_clean":"2024-05-06T00:27:56.716566+0000","last_became_active":"2024-05-06T00:27:21.955822+0000","last_became_peered":"2024-05-06T00:27:21.955822+0000","last_unstale":"2024-05-06T00:27:56.716566+0000","last_undegraded":"2024-05-06T00:27:56.716566+0000","last_fullsized":"2024-05-06T00:27:56.716566+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:15:05.768048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":11875913,"num_objects":39,"num_object_clones":7,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":23,"num_read":5209,"num_read_kb":13535,"num_write":921,"num_write_kb":1516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"75'314","reported_seq":1946,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.885614+0000","last_change":"2024-05-06T00:27:42.622702+0000","last_active":"2024-05-06T00:27:56.885614+0000","last_peered":"2024-05-06T00:27:56.885614+0000","last_clean":"2024-05-06T00:27:56.885614+0000","last_became_active":"2024-05-06T00:27:42.621212+0000","last_became_peered":"2024-05-06T00:27:42.621212+0000","last_unstale":"2024-05-06T00:27:56.885614+0000","last_undegraded":"2024-05-06T00:27:56.885614+0000","last_fullsized":"2024-05-06T00:27:56.885614+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:47:20.958627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":17996926,"num_objects":33,"num_object_clones":5,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":17,"num_read":2937,"num_read_kb":7767,"num_write":420,"num_write_kb":2473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1517694,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"73'143","reported_seq":868,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.739536+0000","last_change":"2024-05-06T00:27:54.711546+0000","last_active":"2024-05-06T00:27:56.739536+0000","last_peered":"2024-05-06T00:27:56.739536+0000","last_clean":"2024-05-06T00:27:56.739536+0000","last_became_active":"2024-05-06T00:27:54.710473+0000","last_became_peered":"2024-05-06T00:27:54.710473+0000","last_unstale":"2024-05-06T00:27:56.739536+0000","last_undegraded":"2024-05-06T00:27:56.739536+0000","last_fullsized":"2024-05-06T00:27:56.739536+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:18:08.964424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":5859156,"num_objects":28,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":15,"num_read":842,"num_read_kb":6946,"num_write":160,"num_write_kb":585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":120488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"75'283","reported_seq":4489,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.850486+0000","last_change":"2024-05-06T00:27:54.717002+0000","last_active":"2024-05-06T00:27:56.850486+0000","last_peered":"2024-05-06T00:27:56.850486+0000","last_clean":"2024-05-06T00:27:56.850486+0000","last_became_active":"2024-05-06T00:27:54.715616+0000","last_became_peered":"2024-05-06T00:27:54.715616+0000","last_unstale":"2024-05-06T00:27:56.850486+0000","last_undegraded":"2024-05-06T00:27:56.850486+0000","last_fullsized":"2024-05-06T00:27:56.850486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:10:43.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":25476096,"num_objects":31,"num_object_clones":4,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":15,"num_read":5583,"num_read_kb":11820,"num_write":188,"num_write_kb":2817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":432128,"num_keys_recovered":34,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"75'150","reported_seq":1103,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.830983+0000","last_change":"2024-05-06T00:27:21.968173+0000","last_active":"2024-05-06T00:27:56.830983+0000","last_peered":"2024-05-06T00:27:56.830983+0000","last_clean":"2024-05-06T00:27:56.830983+0000","last_became_active":"2024-05-06T00:27:21.965833+0000","last_became_peered":"2024-05-06T00:27:21.965833+0000","last_unstale":"2024-05-06T00:27:56.830983+0000","last_undegraded":"2024-05-06T00:27:56.830983+0000","last_fullsized":"2024-05-06T00:27:56.830983+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:09:52.355002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":22814764,"num_objects":33,"num_object_clones":3,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":20,"num_read":970,"num_read_kb":5949,"num_write":178,"num_write_kb":1430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"75'193","reported_seq":3914,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.881934+0000","last_change":"2024-05-06T00:27:54.711712+0000","last_active":"2024-05-06T00:27:56.881934+0000","last_peered":"2024-05-06T00:27:56.881934+0000","last_clean":"2024-05-06T00:27:56.881934+0000","last_became_active":"2024-05-06T00:27:54.710459+0000","last_became_peered":"2024-05-06T00:27:54.710459+0000","last_unstale":"2024-05-06T00:27:56.881934+0000","last_undegraded":"2024-05-06T00:27:56.881934+0000","last_fullsized":"2024-05-06T00:27:56.881934+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:09:03.012622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":31332864,"num_objects":46,"num_object_clones":12,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":23,"num_read":3624,"num_read_kb":13631,"num_write":191,"num_write_kb":2046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":477184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"73'281","reported_seq":6170,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-05-06T00:27:56.714903+0000","last_change":"2024-05-06T00:27:42.615243+0000","last_active":"2024-05-06T00:27:56.714903+0000","last_peered":"2024-05-06T00:27:56.714903+0000","last_clean":"2024-05-06T00:27:56.714903+0000","last_became_active":"2024-05-06T00:27:42.614475+0000","last_became_peered":"2024-05-06T00:27:42.614475+0000","last_unstale":"2024-05-06T00:27:56.714903+0000","last_undegraded":"2024-05-06T00:27:56.714903+0000","last_fullsized":"2024-05-06T00:27:56.714903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:01:55.371583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":26105441,"num_objects":35,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":14,"num_read":6682,"num_read_kb":22988,"num_write":209,"num_write_kb":3244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2163809,"num_keys_recovered":23,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":626738439,"num_objects":1070,"num_object_clones":214,"num_object_copies":2140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":538,"num_read":100360,"num_read_kb":401350,"num_write":9079,"num_write_kb":97931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1108,"num_bytes_recovered":9690413,"num_keys_recovered":241,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":60448768,"data_stored":89529075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8366,"ondisk_log_size":8366,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640277,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57852,"kb_used_data":14192,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94313988,"statfs":{"total":96636764160,"available":96577523712,"internally_reserved":0,"allocated":14532608,"data_stored":22786179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640277,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50608,"kb_used_data":11112,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94321232,"statfs":{"total":96636764160,"available":96584941568,"internally_reserved":0,"allocated":11378688,"data_stored":14754905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51572,"kb_used_data":12076,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94320268,"statfs":{"total":96636764160,"available":96583954432,"internally_reserved":0,"allocated":12365824,"data_stored":15500654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640279,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59904,"kb_used_data":16244,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94311936,"statfs":{"total":96636764160,"available":96575422464,"internally_reserved":0,"allocated":16633856,"data_stored":25022388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":47244640277,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4857856,"data_stored":6773154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15167488,"data_stored":24079016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11493376,"data_stored":15147650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9912320,"data_stored":13811533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13660160,"data_stored":22433175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5357568,"data_stored":7284547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:28:00.593 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-05-06T00:28:00.593 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:00.590+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:00.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:00.594 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:00.594 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-05-06T00:28:00.594 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-05-06T00:28:00.691 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:00.690+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:00.759 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:00.759 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:00.759 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:00.774 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:28:00.775 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:00.821 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:00.819+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:00.893 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:00.894+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:00.897 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone22 from 0x5cc4c00 (overlap 0x2413e00) to 0x0 2024-05-06T00:28:00.900 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:00.900 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:00.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:00.995 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:00.994+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:01.004 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-06T00:28:01.006+0000 7fdfb36a9640 -1 mon.a@0(leader).osd e79 definitely_dead 0 2024-05-06T00:28:01.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:01.096 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:01.091+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:01.102 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.440725626999999, 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:28:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:28:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4614841219999999, 2024-05-06T00:28:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:28:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4614099030000001, 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:28:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:28:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.461348004, 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:28:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4612884049999999, 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:28:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4612042700000001, 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:28:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:28:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4611387570000001, 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.461080895, 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:28:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:28:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4609843, 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:28:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:28:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4609065750000001, 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:28:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:28:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:28:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.460833789, 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:28:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:28:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.460756298, 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:28:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4606621689999999, 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:28:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:28:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4605792069999999, 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:28:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:28:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4605051019999999, 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:28:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:28:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.460405913, 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:28:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:28:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4603427609999999, 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:28:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:28:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.460242496, 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:28:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4601531240000001, 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.2142944410000001, 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:28:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:28:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:01.190 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:01.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:01.197 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:01.198+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:01.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:01.302 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:01.298+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:01.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:01.345 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:01.345 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:01.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:01.390 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:01.390 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:01.390 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:01.390 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:01.398 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:01.395+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:01.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:01.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:01.410 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:01.441 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:01.500 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:01.495+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:01.600 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:01.602+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:01.604 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:01.605 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:01.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:01.702 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:01.702+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:01.761 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:28:01.761 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.0. 2024-05-06T00:28:01.776 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:01.776 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:01.788 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-05-06T00:28:01.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:01.802 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:01.799+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:01.903 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:01.902+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:01.959 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:01.959 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:01.971 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:02.016 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:02.018+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:02.105 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:02.103+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:02.131 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:02.131 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:02.144 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:02.205 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:02.206+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:02.259 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:02.260 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:02.271 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:02.313 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:02.314+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:02.408 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:02.407+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:02.431 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:02.431 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:02.444 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:02.509 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:02.503+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:02.560 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:02.560 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:02.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:02.620 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:02.622+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:02.689 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:02.689 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:02.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:02.710 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:02.707+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:02.811 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:02.810+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:02.888 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:02.889 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:02.901 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:02.912 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:02.907+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:03.013 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:03.011+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:03.053 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:03.053 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:03.066 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:03.113 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:03.114+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:03.185 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:03.185 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:03.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:03.245 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:03.243+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:03.315 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:03.314+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:03.329 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:03.329 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:03.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:03.429 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:03.427+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:03.468 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:03.501 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:03.501 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:03.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:03.557 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:03.555+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:03.618 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:03.618+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:03.622 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:03.680 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:03.681 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:03.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:03.741 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:03.739+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:03.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:28:03.821 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:03.819+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:03.821 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:03.821 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:03.836 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:03.922 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:03.919+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:03.949 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:03.949 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:03.989 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:04.010 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:04.010 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:04.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:04.025 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:04.022+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:04.035 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:04.035 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:04.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:04.061 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:04.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:04.153 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:04.154+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:04.220 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.557709885000001, 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.221 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:04.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5784683800000003, 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.226 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:28:04.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:28:04.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.231 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.231 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:04.231 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5783941610000003, 2024-05-06T00:28:04.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:28:04.231 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.231 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.231 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:04.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:28:04.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.578332262, 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:28:04.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5782726629999999, 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.240 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:28:04.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.244 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5781885280000001, 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.248 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5781230150000001, 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:28:04.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:28:04.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:28:04.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:28:04.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:28:04.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:28:04.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5780651529999998, 2024-05-06T00:28:04.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:28:04.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5779685580000002, 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.259 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:28:04.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.263 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5778908329999997, 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.264 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:04.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.268 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5778180470000001, 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:04.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.272 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5777405560000002, 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.273 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:28:04.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:28:04.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:04.277 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5776464270000004, 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5775634649999999, 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:28:04.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:28:04.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5774893599999995, 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.287 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5773901710000002, 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:28:04.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5773270190000002, 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.297 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:28:04.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5772267539999998, 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.301 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.305 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5771373820000001, 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:28:04.309 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.3312786990000003, 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:28:04.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.314 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:04.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:04.316 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:04.311+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:04.417 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:04.415+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:04.504 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:04.504 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:04.518 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:04.518+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:04.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:04.619 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:04.615+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:04.701 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:04.701 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:04.719 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:04.718+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:04.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:04.796 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:28:04.821 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:04.819+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:04.905 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.444601757999997, 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.906 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:28:04.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:28:04.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:04.910 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.443799769000002, 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:04.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.914 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.443713314999997, 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.915 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:28:04.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:28:04.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:28:04.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.443640721999998, 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.443580871000002, 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.923 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:04.927 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.443505188000003, 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:04.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:28:04.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2967 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 159744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.435881199999997, 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026274390000000002, 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2967 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467898+0000", 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.046e-06 2024-05-06T00:28:04.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467917+0000", 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467937+0000", 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9613999999999999e-05 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467961+0000", 2024-05-06T00:28:04.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.419e-05 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.493935+0000", 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025973922 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494053+0000", 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011866099999999999 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494171+0000", 2024-05-06T00:28:04.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011806099999999999 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.435799615000001, 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:28:04.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.435745408000002, 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:04.944 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.435693258999997, 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:04.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.948 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3040 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.433837937, 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026298018999999999, 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.949 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3040 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469941+0000", 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0979999999999999e-06 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469946+0000", 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0170000000000004e-06 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469963+0000", 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6541999999999998e-05 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469977+0000", 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4015999999999999e-05 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495954+0000", 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025976902999999999 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496101+0000", 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147529 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496238+0000", 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136914 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.433735544000001, 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:28:04.953 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:28:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.957 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.433659204000001, 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:28:04.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:28:04.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.618521497000003, 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.962 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:04.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.618469056000002, 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.968 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:28:04.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:04.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:04.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:04.973 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.611449073000003, 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:04.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:28:04.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:28:04.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:28:04.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.611402362, 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:28:04.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.987 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.611072254, 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:28:04.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.611044331999999, 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.993 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:04.997 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.600177443, 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:04.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:04.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:05.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:05.001 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:05.003 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:04.999+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:05.003 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:05.103 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:05.102+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:05.169 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11628 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802437+0000", 2024-05-06T00:28:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34041777699999998, 2024-05-06T00:28:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34045796499999997, 2024-05-06T00:28:05.169 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11628 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802437+0000", 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802437+0000", 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802437+0000", 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802440+0000", 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6620000000000001e-06 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802446+0000", 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9630000000000003e-06 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802451+0000", 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6019999999999999e-06 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803375+0000", 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00092365100000000005 2024-05-06T00:28:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803380+0000", 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-06 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805930+0000", 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00255017 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805936+0000", 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7350000000000001e-06 2024-05-06T00:28:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33214 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804187+0000", 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33866786399999999, 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33882130999999999, 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33214 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804187+0000", 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804187+0000", 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804187+0000", 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1899999999999998e-07 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804205+0000", 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7614000000000001e-05 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804216+0000", 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.102e-05 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804221+0000", 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7400000000000001e-06 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804454+0000", 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023259100000000001 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804460+0000", 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5940000000000001e-06 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807438+0000", 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002978343 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807445+0000", 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6730000000000003e-06 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33365 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805187+0000", 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33766691399999998, 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33792914400000001, 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33365 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805187+0000", 2024-05-06T00:28:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805187+0000", 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805188+0000", 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.03e-07 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805190+0000", 2024-05-06T00:28:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805197+0000", 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7920000000000004e-06 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805202+0000", 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2379999999999997e-06 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808410+0000", 2024-05-06T00:28:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032078879999999999 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808415+0000", 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3759999999999999e-06 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33442 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.807080+0000", 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33577425799999999, 2024-05-06T00:28:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33610856700000002, 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33442 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807080+0000", 2024-05-06T00:28:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807080+0000", 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807081+0000", 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8799999999999998e-07 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807083+0000", 2024-05-06T00:28:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.17e-06 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807089+0000", 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8209999999999998e-06 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807099+0000", 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0118999999999999e-05 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809360+0000", 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022614559999999998 2024-05-06T00:28:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809365+0000", 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3720000000000004e-06 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33579 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810296+0000", 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33255861399999997, 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33296277200000002, 2024-05-06T00:28:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33579 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810296+0000", 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810296+0000", 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810296+0000", 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810304+0000", 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4240000000000004e-06 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810309+0000", 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7389999999999996e-06 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810314+0000", 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6959999999999996e-06 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812876+0000", 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002562445 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812882+0000", 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33632 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810992+0000", 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33186270200000001, 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.332335832, 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33632 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810992+0000", 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810992+0000", 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810992+0000", 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7399999999999999e-07 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810994+0000", 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.12e-06 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811001+0000", 2024-05-06T00:28:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3319999999999997e-06 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811005+0000", 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8369999999999996e-06 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813880+0000", 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028745789999999999 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813885+0000", 2024-05-06T00:28:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2050000000000001e-06 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33946 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811639+0000", 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33121545000000002, 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.331756683, 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33946 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811639+0000", 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811639+0000", 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811639+0000", 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4799999999999999e-07 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811641+0000", 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811647+0000", 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.913e-06 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811652+0000", 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2020000000000003e-06 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816253+0000", 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046002600000000001 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816260+0000", 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9110000000000004e-06 2024-05-06T00:28:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32994567200000002, 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33055726699999999, 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:28:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:28:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11629 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802457+0000", 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340397018, 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.341077091, 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11629 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802457+0000", 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802457+0000", 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802458+0000", 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.77e-07 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802461+0000", 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.768e-06 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802467+0000", 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3280000000000002e-06 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802472+0000", 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0640000000000001e-06 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803403+0000", 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00093084500000000002 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803407+0000", 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6890000000000003e-06 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805960+0000", 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025527290000000001 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805966+0000", 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3530000000000002e-06 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33217 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804244+0000", 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33861038999999998, 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33937031600000001, 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33217 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804244+0000", 2024-05-06T00:28:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804244+0000", 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804245+0000", 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3499999999999996e-07 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804247+0000", 2024-05-06T00:28:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4700000000000001e-06 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804254+0000", 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0670000000000003e-06 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804274+0000", 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9494999999999999e-05 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804504+0000", 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023070299999999999 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804508+0000", 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5920000000000001e-06 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807468+0000", 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029602690000000002 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807474+0000", 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3219999999999998e-06 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33367 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805209+0000", 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33764567200000001, 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33848573300000001, 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33367 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805209+0000", 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805209+0000", 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805209+0000", 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805211+0000", 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1730000000000002e-06 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805218+0000", 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3110000000000002e-06 2024-05-06T00:28:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805222+0000", 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5260000000000004e-06 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808439+0000", 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032170979999999998 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808445+0000", 2024-05-06T00:28:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4380000000000003e-06 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33444 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.807106+0000", 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33574836000000002, 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33665645100000002, 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33444 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807106+0000", 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807106+0000", 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807106+0000", 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807109+0000", 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4370000000000001e-06 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807115+0000", 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3629999999999999e-06 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807120+0000", 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5299999999999998e-06 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809389+0000", 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022693240000000001 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809394+0000", 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2959999999999998e-06 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33580 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810320+0000", 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33253458600000002, 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33358326399999999, 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33580 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810320+0000", 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810320+0000", 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810320+0000", 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2800000000000003e-07 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810322+0000", 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8470000000000001e-06 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810327+0000", 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2939999999999996e-06 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810332+0000", 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8269999999999997e-06 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812908+0000", 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025759960000000001 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812913+0000", 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3900000000000001e-06 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33634 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811012+0000", 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.331842786, 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33300569899999999, 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33634 2024-05-06T00:28:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811012+0000", 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811012+0000", 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811012+0000", 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5600000000000001e-07 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811014+0000", 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0169999999999999e-06 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811020+0000", 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4710000000000004e-06 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811025+0000", 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8239999999999999e-06 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813906+0000", 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028811079999999998 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813911+0000", 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5600000000000004e-06 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33948 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811658+0000", 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33119647299999999, 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.332429857, 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33948 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811658+0000", 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811658+0000", 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811658+0000", 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3000000000000002e-07 2024-05-06T00:28:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811660+0000", 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.843e-06 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811666+0000", 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.49e-06 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811670+0000", 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4220000000000002e-06 2024-05-06T00:28:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816282+0000", 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046121000000000001 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816289+0000", 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8059999999999997e-06 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.329919395, 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33122323199999998, 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11630 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802479+0000", 2024-05-06T00:28:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34037579499999998, 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34174790300000002, 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11630 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802479+0000", 2024-05-06T00:28:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802479+0000", 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802479+0000", 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1399999999999997e-07 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802482+0000", 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0570000000000001e-06 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802488+0000", 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0469999999999999e-06 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802493+0000", 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0479999999999998e-06 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803431+0000", 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00093769199999999995 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803436+0000", 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9189999999999998e-06 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805990+0000", 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002554167 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805995+0000", 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3519999999999997e-06 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33219 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804306+0000", 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33854865099999998, 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34000079900000002, 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33219 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804306+0000", 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804306+0000", 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804306+0000", 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4899999999999995e-07 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804309+0000", 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.083e-06 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804324+0000", 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4980999999999999e-05 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804331+0000", 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.336e-06 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804532+0000", 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020101099999999999 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804534+0000", 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6309999999999999e-06 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807500+0000", 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002965278 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807505+0000", 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5890000000000002e-06 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33369 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805228+0000", 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337626172, 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33915771500000003, 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33369 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805228+0000", 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805228+0000", 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805229+0000", 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2099999999999998e-07 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805231+0000", 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9800000000000001e-06 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805237+0000", 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9939999999999997e-06 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805242+0000", 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0989999999999998e-06 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808469+0000", 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032270739999999999 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808474+0000", 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3619999999999997e-06 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33446 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.807126+0000", 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33572876200000001, 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33732825799999999, 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33446 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807126+0000", 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807126+0000", 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807126+0000", 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1e-07 2024-05-06T00:28:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807128+0000", 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.885e-06 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807133+0000", 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9370000000000003e-06 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807145+0000", 2024-05-06T00:28:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2131e-05 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809676+0000", 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025307229999999999 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809682+0000", 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3690000000000003e-06 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33581 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810338+0000", 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33251597999999999, 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.334185381, 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33581 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810338+0000", 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810338+0000", 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810339+0000", 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8500000000000002e-07 2024-05-06T00:28:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810341+0000", 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1299999999999999e-06 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810346+0000", 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5249999999999996e-06 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810351+0000", 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7029999999999997e-06 2024-05-06T00:28:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812939+0000", 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587724 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812944+0000", 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3589999999999999e-06 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33636 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811032+0000", 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33182288300000001, 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33356006700000002, 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33636 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811032+0000", 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811032+0000", 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811032+0000", 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811034+0000", 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811041+0000", 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.72e-06 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811045+0000", 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7929999999999997e-06 2024-05-06T00:28:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813933+0000", 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028875519999999998 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813938+0000", 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33950 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811675+0000", 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33117912199999999, 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33298386600000002, 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33950 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811675+0000", 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811675+0000", 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811676+0000", 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2300000000000002e-07 2024-05-06T00:28:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811677+0000", 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8870000000000001e-06 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811683+0000", 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4489999999999998e-06 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811688+0000", 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7690000000000004e-06 2024-05-06T00:28:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816312+0000", 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046239699999999998 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816318+0000", 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6309999999999997e-06 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32987615399999998, 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.331759211, 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11631 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802500+0000", 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34035468299999999, 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34231862400000002, 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11631 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802500+0000", 2024-05-06T00:28:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802500+0000", 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802500+0000", 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3000000000000001e-07 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802509+0000", 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6680000000000007e-06 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802516+0000", 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9720000000000003e-06 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802521+0000", 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2920000000000003e-06 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803460+0000", 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00093890300000000005 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803465+0000", 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.848e-06 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806016+0000", 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025514890000000001 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806023+0000", 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9770000000000003e-06 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33307 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804346+0000", 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33850802800000002, 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34058196400000001, 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33307 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804346+0000", 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804346+0000", 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804347+0000", 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2300000000000001e-07 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804350+0000", 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8009999999999999e-06 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804363+0000", 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3069e-05 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804369+0000", 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2110000000000003e-06 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804550+0000", 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000181161 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804553+0000", 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1429999999999998e-06 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807533+0000", 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002979256 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807538+0000", 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1209999999999998e-06 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33371 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805248+0000", 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33760622200000001, 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33984053400000003, 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33371 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805248+0000", 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805248+0000", 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805249+0000", 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0400000000000002e-07 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805251+0000", 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1849999999999998e-06 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805257+0000", 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0499999999999997e-06 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805261+0000", 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4909999999999999e-06 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808498+0000", 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003236779 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808503+0000", 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2379999999999997e-06 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.282 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33448 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.807151+0000", 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33570298900000001, 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.338009438, 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33448 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807151+0000", 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807151+0000", 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807152+0000", 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807154+0000", 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0729999999999999e-06 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807161+0000", 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7569999999999998e-06 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807165+0000", 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7400000000000004e-06 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811931+0000", 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047654280000000004 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811938+0000", 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7349999999999999e-06 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33582 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810357+0000", 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33249736000000002, 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33487485500000003, 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33582 2024-05-06T00:28:05.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810357+0000", 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810357+0000", 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810357+0000", 2024-05-06T00:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.53e-07 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810360+0000", 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2299999999999998e-06 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810367+0000", 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1890000000000001e-06 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810371+0000", 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5820000000000003e-06 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812980+0000", 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026088330000000001 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812986+0000", 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.711e-06 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33912 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811052+0000", 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.331802506, 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33424784600000002, 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33912 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.291 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811052+0000", 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811052+0000", 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811052+0000", 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8299999999999998e-07 2024-05-06T00:28:05.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811054+0000", 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1430000000000001e-06 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811061+0000", 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7109999999999998e-06 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811066+0000", 2024-05-06T00:28:05.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8790000000000002e-06 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813964+0000", 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028983239999999999 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813970+0000", 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1989999999999997e-06 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33951 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811694+0000", 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33116059199999998, 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33367779199999997, 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33951 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.295 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811694+0000", 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811694+0000", 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811694+0000", 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7399999999999999e-07 2024-05-06T00:28:05.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811696+0000", 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9800000000000001e-06 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811702+0000", 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5430000000000001e-06 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811708+0000", 2024-05-06T00:28:05.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816341+0000", 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004633585 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816348+0000", 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7649999999999996e-06 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.329846525, 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.332431527, 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.299 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:05.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:28:05.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11632 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802528+0000", 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34032605799999999, 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.342982173, 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11632 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802528+0000", 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802528+0000", 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802529+0000", 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0500000000000004e-07 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802532+0000", 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8049999999999998e-06 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802538+0000", 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.46e-06 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802543+0000", 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:28:05.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803489+0000", 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00094599699999999996 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803494+0000", 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806044+0000", 2024-05-06T00:28:05.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002549791 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806049+0000", 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5570000000000003e-06 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33309 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804378+0000", 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33847686199999999, 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34121592299999998, 2024-05-06T00:28:05.307 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33309 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804378+0000", 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804378+0000", 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804378+0000", 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7e-07 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804380+0000", 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804387+0000", 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8340000000000001e-06 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804393+0000", 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3229999999999997e-06 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804569+0000", 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017657299999999999 2024-05-06T00:28:05.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804573+0000", 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.721e-06 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807567+0000", 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029944049999999999 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807573+0000", 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6969999999999998e-06 2024-05-06T00:28:05.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33373 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805267+0000", 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33758701699999999, 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34040624400000002, 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33373 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805267+0000", 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805267+0000", 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805268+0000", 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1300000000000001e-07 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805284+0000", 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6299000000000001e-05 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805291+0000", 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3189999999999997e-06 2024-05-06T00:28:05.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805297+0000", 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3029999999999999e-06 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808527+0000", 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032307590000000001 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808533+0000", 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2839999999999997e-06 2024-05-06T00:28:05.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33450 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809703+0000", 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33315138399999999, 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.336041851, 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33450 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809703+0000", 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809703+0000", 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809704+0000", 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.215e-06 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809709+0000", 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9710000000000003e-06 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809718+0000", 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7129999999999994e-06 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809725+0000", 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7660000000000001e-06 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811959+0000", 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022347130000000002 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811965+0000", 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2809999999999999e-06 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33583 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810378+0000", 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.332476628, 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33543723399999997, 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33583 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810378+0000", 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810378+0000", 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810378+0000", 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6300000000000001e-07 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810380+0000", 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3499999999999999e-06 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810388+0000", 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810392+0000", 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9529999999999998e-06 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813012+0000", 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002619442 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813018+0000", 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6890000000000001e-06 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33913 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811072+0000", 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33178234899999998, 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33481104, 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33913 2024-05-06T00:28:05.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811072+0000", 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811072+0000", 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811073+0000", 2024-05-06T00:28:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6599999999999996e-07 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811079+0000", 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9630000000000003e-06 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811086+0000", 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2880000000000004e-06 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811092+0000", 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6489999999999996e-06 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813992+0000", 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029003869999999999 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813997+0000", 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9599999999999999e-06 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33954 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811714+0000", 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33114041500000002, 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33424171699999999, 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33954 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.328 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811714+0000", 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811714+0000", 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811714+0000", 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7300000000000002e-07 2024-05-06T00:28:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811716+0000", 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811723+0000", 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6120000000000004e-06 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811728+0000", 2024-05-06T00:28:05.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816371+0000", 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046429139999999997 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816377+0000", 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8000000000000001e-06 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32981367499999997, 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33298467199999998, 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:05.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.335 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11633 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802550+0000", 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34030475199999999, 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34355792200000002, 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11633 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802550+0000", 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802550+0000", 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802550+0000", 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802553+0000", 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.01e-06 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802559+0000", 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2570000000000004e-06 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802565+0000", 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4569999999999996e-06 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803517+0000", 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095241499999999997 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803522+0000", 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9910000000000002e-06 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806074+0000", 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002551827 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806079+0000", 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3449999999999996e-06 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33311 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804399+0000", 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.338455269, 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.341791701, 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33311 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804399+0000", 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804399+0000", 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804400+0000", 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804402+0000", 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0499999999999999e-06 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804408+0000", 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804413+0000", 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6550000000000003e-06 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804588+0000", 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017527100000000001 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804592+0000", 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1450000000000001e-06 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807600+0000", 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030079299999999998 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807606+0000", 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3439999999999997e-06 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33374 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805305+0000", 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337549194, 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.341042393, 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33374 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.345 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805305+0000", 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805305+0000", 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805306+0000", 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.51e-07 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805308+0000", 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805315+0000", 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4309999999999997e-06 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805321+0000", 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2979999999999999e-06 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808557+0000", 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032360269999999998 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808562+0000", 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2619999999999998e-06 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33452 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809737+0000", 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33311774799999999, 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33671187200000002, 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33452 2024-05-06T00:28:05.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809737+0000", 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809737+0000", 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809737+0000", 2024-05-06T00:28:05.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4499999999999997e-07 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809740+0000", 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5280000000000002e-06 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809746+0000", 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2009999999999996e-06 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809751+0000", 2024-05-06T00:28:05.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0549999999999999e-06 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811987+0000", 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022365570000000001 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811993+0000", 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4e-06 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33584 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810399+0000", 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33245545500000001, 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33612038999999999, 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33584 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810399+0000", 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810399+0000", 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810399+0000", 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1e-07 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810401+0000", 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1289999999999999e-06 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810407+0000", 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0889999999999996e-06 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810412+0000", 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.775e-06 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813045+0000", 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026323760000000001 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813050+0000", 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5470000000000004e-06 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33914 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.356 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811101+0000", 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33175379300000002, 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.335492391, 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33914 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811101+0000", 2024-05-06T00:28:05.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811101+0000", 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811101+0000", 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811103+0000", 2024-05-06T00:28:05.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811108+0000", 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2059999999999998e-06 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811113+0000", 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814020+0000", 2024-05-06T00:28:05.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029064339999999998 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814025+0000", 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33956 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811734+0000", 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33112050799999998, 2024-05-06T00:28:05.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33492994500000001, 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33956 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811734+0000", 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811734+0000", 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811734+0000", 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3500000000000002e-07 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811736+0000", 2024-05-06T00:28:05.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.21e-06 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811742+0000", 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.784e-06 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811747+0000", 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.318e-06 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816459+0000", 2024-05-06T00:28:05.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047124389999999997 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816467+0000", 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8169999999999999e-06 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32978203699999997, 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33365975199999998, 2024-05-06T00:28:05.364 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33181 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802562+0000", 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340291974, 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.344240875, 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33181 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802562+0000", 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802562+0000", 2024-05-06T00:28:05.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802563+0000", 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8299999999999995e-07 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802567+0000", 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.861e-06 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802579+0000", 2024-05-06T00:28:05.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1875000000000001e-05 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802605+0000", 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5466e-05 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802695+0000", 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0148999999999994e-05 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802701+0000", 2024-05-06T00:28:05.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8819999999999998e-06 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806513+0000", 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038125569999999998 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806519+0000", 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4439999999999999e-06 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33313 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804419+0000", 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33843546899999999, 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34248329399999999, 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33313 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804419+0000", 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804419+0000", 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804419+0000", 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2800000000000003e-07 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804421+0000", 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.818e-06 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804426+0000", 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2020000000000003e-06 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804430+0000", 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1659999999999996e-06 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804606+0000", 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000175382 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804610+0000", 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6659999999999999e-06 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807629+0000", 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030187899999999999 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807635+0000", 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.716e-06 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33376 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805328+0000", 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337526877, 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34165978600000002, 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33376 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805328+0000", 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805328+0000", 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805328+0000", 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000002e-07 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805330+0000", 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805337+0000", 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4010000000000002e-06 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805343+0000", 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0939999999999996e-06 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808586+0000", 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003243185 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808591+0000", 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3759999999999999e-06 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33453 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809758+0000", 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33309673499999998, 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.337301395, 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33453 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809758+0000", 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809758+0000", 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809758+0000", 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809760+0000", 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809765+0000", 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9030000000000003e-06 2024-05-06T00:28:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809769+0000", 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9600000000000002e-06 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812033+0000", 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022642130000000002 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812039+0000", 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.694e-06 2024-05-06T00:28:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33585 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810419+0000", 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33243575199999997, 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33670831800000001, 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33585 2024-05-06T00:28:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810419+0000", 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810419+0000", 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810419+0000", 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810421+0000", 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.255e-06 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810428+0000", 2024-05-06T00:28:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9999999999999999e-06 2024-05-06T00:28:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810433+0000", 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7199999999999997e-06 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002642288 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813081+0000", 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6019999999999999e-06 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33915 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811120+0000", 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33173449700000002, 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33612866299999999, 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33915 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811120+0000", 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811120+0000", 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811120+0000", 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5699999999999998e-07 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811122+0000", 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1059999999999998e-06 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811129+0000", 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.144e-06 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811133+0000", 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814049+0000", 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029152610000000002 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814053+0000", 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33958 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811753+0000", 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.331101743, 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33559680200000003, 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33958 2024-05-06T00:28:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811753+0000", 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811753+0000", 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811753+0000", 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7300000000000002e-07 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811755+0000", 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811761+0000", 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7169999999999996e-06 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811765+0000", 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.651e-06 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816499+0000", 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047334660000000004 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816509+0000", 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0433e-05 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32974923499999997, 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33433008199999997, 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:28:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:28:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11634 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802572+0000", 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34028278099999998, 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.344932242, 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11634 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802572+0000", 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802572+0000", 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802572+0000", 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1299999999999999e-07 2024-05-06T00:28:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802575+0000", 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8049999999999998e-06 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802581+0000", 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5910000000000003e-06 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802586+0000", 2024-05-06T00:28:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.809e-06 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803546+0000", 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096054799999999998 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803551+0000", 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806101+0000", 2024-05-06T00:28:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025494799999999998 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806106+0000", 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.366e-06 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33315 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804454+0000", 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33840065600000002, 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34312988100000003, 2024-05-06T00:28:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33315 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804454+0000", 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804454+0000", 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804454+0000", 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.77e-07 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804457+0000", 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4880000000000001e-06 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804463+0000", 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0990000000000004e-06 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804467+0000", 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7929999999999997e-06 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804636+0000", 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016892 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804642+0000", 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7200000000000003e-06 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807659+0000", 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003016679 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807664+0000", 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3920000000000002e-06 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33378 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805349+0000", 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33750527800000002, 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34231439499999999, 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33378 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805349+0000", 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805349+0000", 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805349+0000", 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6600000000000002e-07 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805352+0000", 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0490000000000002e-06 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805357+0000", 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8599999999999998e-06 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805362+0000", 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808615+0000", 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032533240000000001 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808621+0000", 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.339e-06 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33455 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809775+0000", 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33307937500000001, 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33795775700000003, 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33455 2024-05-06T00:28:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809775+0000", 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809775+0000", 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809775+0000", 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1699999999999999e-07 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809777+0000", 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1289999999999999e-06 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809783+0000", 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6860000000000003e-06 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809787+0000", 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0099999999999997e-06 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812072+0000", 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022853209999999999 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812078+0000", 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5620000000000003e-06 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33586 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810439+0000", 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.332414979, 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33736118900000001, 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33586 2024-05-06T00:28:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810439+0000", 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810439+0000", 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810440+0000", 2024-05-06T00:28:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7099999999999997e-07 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810442+0000", 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3099999999999999e-06 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810449+0000", 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.567e-06 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810454+0000", 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8520000000000003e-06 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026546669999999999 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813113+0000", 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2839999999999997e-06 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33916 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811140+0000", 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33171473299999998, 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.336728743, 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33916 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811140+0000", 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811140+0000", 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811140+0000", 2024-05-06T00:28:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7099999999999997e-07 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811142+0000", 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0420000000000001e-06 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811148+0000", 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1369999999999999e-06 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811153+0000", 2024-05-06T00:28:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814077+0000", 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029237310000000002 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814082+0000", 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4550000000000003e-06 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33960 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811771+0000", 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33108304, 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33616480300000001, 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33960 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811771+0000", 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811771+0000", 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811772+0000", 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5600000000000001e-07 2024-05-06T00:28:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811774+0000", 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.95e-06 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811780+0000", 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8000000000000001e-06 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811785+0000", 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816540+0000", 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047544859999999996 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816547+0000", 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32971720199999999, 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.334866734, 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11608 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.801953+0000", 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34090129699999999, 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34611895100000001, 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11608 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.801953+0000", 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.801953+0000", 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.801955+0000", 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.722e-06 2024-05-06T00:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.801963+0000", 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8949999999999998e-06 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.801976+0000", 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3712e-05 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.801985+0000", 2024-05-06T00:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7819999999999999e-06 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802817+0000", 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00083221200000000003 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802825+0000", 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805367+0000", 2024-05-06T00:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025429659999999998 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805375+0000", 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11635 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802592+0000", 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34026240000000002, 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34555997999999999, 2024-05-06T00:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11635 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802592+0000", 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802592+0000", 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802592+0000", 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9999999999999998e-07 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802598+0000", 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0580000000000003e-06 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802605+0000", 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1360000000000002e-06 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802609+0000", 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5839999999999996e-06 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803575+0000", 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096610499999999998 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803580+0000", 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.831e-06 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806127+0000", 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025472450000000001 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806133+0000", 2024-05-06T00:28:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9200000000000001e-06 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33317 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804476+0000", 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33837861600000002, 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34380949300000002, 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33317 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804476+0000", 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804476+0000", 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804476+0000", 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1100000000000002e-07 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804478+0000", 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1749999999999999e-06 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804484+0000", 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9649999999999996e-06 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804488+0000", 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1250000000000003e-06 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804730+0000", 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024128499999999999 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804735+0000", 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.344e-06 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807690+0000", 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029553829999999998 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807696+0000", 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4689999999999996e-06 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33380 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805369+0000", 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337485747, 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34301424200000002, 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33380 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805369+0000", 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805369+0000", 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805369+0000", 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805371+0000", 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3410000000000001e-06 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805378+0000", 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0489999999999998e-06 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805384+0000", 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0690000000000001e-06 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808644+0000", 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032607780000000002 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808650+0000", 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3940000000000004e-06 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33457 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809793+0000", 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.333061836, 2024-05-06T00:28:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.3386593, 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33457 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809793+0000", 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809793+0000", 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809793+0000", 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809820+0000", 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6927000000000002e-05 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809826+0000", 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6309999999999997e-06 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809831+0000", 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7570000000000004e-06 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812106+0000", 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022744839999999998 2024-05-06T00:28:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812111+0000", 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33587 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810460+0000", 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33239417900000001, 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33805982899999998, 2024-05-06T00:28:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33587 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810460+0000", 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810460+0000", 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810461+0000", 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999998e-07 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810463+0000", 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5469999999999999e-06 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810469+0000", 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6620000000000002e-06 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810474+0000", 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.848e-06 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813141+0000", 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026673619999999999 2024-05-06T00:28:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813147+0000", 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7089999999999999e-06 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33917 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811159+0000", 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33169517199999998, 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.337439922, 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33917 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811159+0000", 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811159+0000", 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811160+0000", 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811162+0000", 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0849999999999999e-06 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811169+0000", 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811174+0000", 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0350000000000001e-06 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814107+0000", 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029330099999999998 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814112+0000", 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5949999999999998e-06 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33962 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811792+0000", 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33106243400000002, 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33687544600000002, 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33962 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811792+0000", 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811792+0000", 2024-05-06T00:28:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811793+0000", 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811795+0000", 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0389999999999999e-06 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811913+0000", 2024-05-06T00:28:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118281 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811920+0000", 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5750000000000003e-06 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816578+0000", 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046577190000000003 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816586+0000", 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3729999999999993e-06 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32968512300000002, 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33556649, 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11609 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802010+0000", 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340844492, 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34679373400000002, 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11609 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802010+0000", 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802010+0000", 2024-05-06T00:28:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802010+0000", 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8500000000000001e-07 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802015+0000", 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0369999999999996e-06 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802023+0000", 2024-05-06T00:28:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0830000000000004e-06 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802028+0000", 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.806e-06 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802851+0000", 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00082238999999999999 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802856+0000", 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2650000000000004e-06 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805399+0000", 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002543006 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805405+0000", 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5149999999999997e-06 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11636 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802616+0000", 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34023800100000001, 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34626691300000001, 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11636 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802616+0000", 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802616+0000", 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802617+0000", 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-06T00:28:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802620+0000", 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1049999999999999e-06 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802626+0000", 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.302e-06 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802631+0000", 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9010000000000002e-06 2024-05-06T00:28:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803603+0000", 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00097236100000000004 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803608+0000", 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8559999999999998e-06 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806155+0000", 2024-05-06T00:28:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002547049 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806163+0000", 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3030000000000002e-06 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33319 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804493+0000", 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33836183600000003, 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34447035999999998, 2024-05-06T00:28:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33319 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804493+0000", 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804493+0000", 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804493+0000", 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2000000000000001e-07 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804494+0000", 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3540000000000001e-06 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804497+0000", 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2780000000000002e-06 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804500+0000", 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.593e-06 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804761+0000", 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026118000000000002 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804766+0000", 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8060000000000002e-06 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807719+0000", 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029526050000000001 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807724+0000", 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33382 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805390+0000", 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33746487200000003, 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34365374599999998, 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33382 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805390+0000", 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805390+0000", 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805390+0000", 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805392+0000", 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9520000000000001e-06 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805397+0000", 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2950000000000001e-06 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805402+0000", 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8609999999999997e-06 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808674+0000", 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032719429999999998 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808679+0000", 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1819999999999997e-06 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33459 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809839+0000", 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33301538600000002, 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33927190400000001, 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.497 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33459 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809839+0000", 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809839+0000", 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809839+0000", 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4700000000000002e-07 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809842+0000", 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0729999999999999e-06 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809847+0000", 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3299999999999998e-06 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809851+0000", 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3900000000000003e-06 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812139+0000", 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022877819999999999 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812145+0000", 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5979999999999996e-06 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33588 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810480+0000", 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33237414500000001, 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33871907699999998, 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.501 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33588 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810480+0000", 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810480+0000", 2024-05-06T00:28:05.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810481+0000", 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999998e-07 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810483+0000", 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1799999999999999e-06 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810490+0000", 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8839999999999997e-06 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810495+0000", 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0100000000000003e-06 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026777540000000001 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813178+0000", 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6200000000000004e-06 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33918 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811180+0000", 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33167397999999998, 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33808872299999998, 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33918 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811180+0000", 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811180+0000", 2024-05-06T00:28:05.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811181+0000", 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9200000000000002e-07 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811183+0000", 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3379999999999999e-06 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811190+0000", 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7229999999999998e-06 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811195+0000", 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8899999999999998e-06 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814137+0000", 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002942676 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814143+0000", 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.789e-06 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33964 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811939+0000", 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33091507100000001, 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33742565800000002, 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33964 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811939+0000", 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811939+0000", 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811940+0000", 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6499999999999999e-07 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811944+0000", 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8999999999999999e-06 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811952+0000", 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1429999999999998e-06 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811958+0000", 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7729999999999996e-06 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816614+0000", 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046563519999999999 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816622+0000", 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5839999999999997e-06 2024-05-06T00:28:05.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33174 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802010+0000", 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340844647, 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34744130000000001, 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33174 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802010+0000", 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802010+0000", 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802011+0000", 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4780000000000001e-06 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802018+0000", 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3339999999999998e-06 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802038+0000", 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0587000000000001e-05 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802046+0000", 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3100000000000001e-06 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802135+0000", 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8321999999999999e-05 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802142+0000", 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4900000000000003e-06 2024-05-06T00:28:05.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806396+0000", 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042535680000000001 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806403+0000", 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6320000000000008e-06 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11637 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802638+0000", 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34021646300000002, 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34689344999999999, 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11637 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.518 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802638+0000", 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802638+0000", 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802639+0000", 2024-05-06T00:28:05.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802641+0000", 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6749999999999998e-06 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802647+0000", 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4739999999999996e-06 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802651+0000", 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3250000000000001e-06 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803631+0000", 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00097998599999999996 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803636+0000", 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2009999999999998e-06 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806184+0000", 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025481660000000001 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806191+0000", 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.64e-06 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33321 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804532+0000", 2024-05-06T00:28:05.522 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33832214199999999, 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34507877599999998, 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33321 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804532+0000", 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804532+0000", 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804533+0000", 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3200000000000005e-07 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804536+0000", 2024-05-06T00:28:05.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.309e-06 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804546+0000", 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4199999999999996e-06 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804553+0000", 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4410000000000004e-06 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804788+0000", 2024-05-06T00:28:05.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023489 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804793+0000", 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3900000000000001e-06 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807748+0000", 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029549189999999999 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807753+0000", 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0579999999999997e-06 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33384 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805408+0000", 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33744642600000002, 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34428314500000001, 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33384 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805408+0000", 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805408+0000", 2024-05-06T00:28:05.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805408+0000", 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9999999999999998e-07 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805411+0000", 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3630000000000001e-06 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805416+0000", 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6980000000000003e-06 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805421+0000", 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.549e-06 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808703+0000", 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032822720000000001 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808709+0000", 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3120000000000001e-06 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33461 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809857+0000", 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33299694800000001, 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33990120499999998, 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.531 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33461 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809857+0000", 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809857+0000", 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809858+0000", 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809860+0000", 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0099999999999998e-06 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809865+0000", 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4550000000000003e-06 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809869+0000", 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0539999999999996e-06 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812173+0000", 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023033950000000002 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812178+0000", 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5350000000000004e-06 2024-05-06T00:28:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33589 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810501+0000", 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33235316599999998, 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33932473499999999, 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33589 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810501+0000", 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810501+0000", 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810502+0000", 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.96e-07 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810504+0000", 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.334e-06 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810510+0000", 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7740000000000001e-06 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810514+0000", 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7570000000000004e-06 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813204+0000", 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026893870000000001 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813209+0000", 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4559999999999999e-06 2024-05-06T00:28:05.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33919 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811201+0000", 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.331653109, 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33869191399999998, 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33919 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811201+0000", 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811201+0000", 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811202+0000", 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811204+0000", 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9599999999999999e-06 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811210+0000", 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9699999999999996e-06 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811214+0000", 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7759999999999997e-06 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814502+0000", 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032873630000000002 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814509+0000", 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33966 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811965+0000", 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33088926000000002, 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33799533300000001, 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33966 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811965+0000", 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811965+0000", 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811966+0000", 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-06T00:28:05.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811968+0000", 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2749999999999998e-06 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811974+0000", 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6740000000000002e-06 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811978+0000", 2024-05-06T00:28:05.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2939999999999999e-06 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816649+0000", 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046708339999999996 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816656+0000", 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9290000000000001e-06 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11610 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802037+0000", 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34081716099999998, 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34799132300000002, 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11610 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802037+0000", 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802037+0000", 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802038+0000", 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802041+0000", 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2090000000000001e-06 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802048+0000", 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0929999999999997e-06 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802054+0000", 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5990000000000001e-06 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802879+0000", 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00082524599999999997 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802884+0000", 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1449999999999999e-06 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805427+0000", 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002542402 2024-05-06T00:28:05.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805432+0000", 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4530000000000001e-06 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33182 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802620+0000", 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340234327, 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34748778699999999, 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33182 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802620+0000", 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802620+0000", 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802621+0000", 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8100000000000002e-07 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802636+0000", 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4846e-05 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802654+0000", 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8513999999999999e-05 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802667+0000", 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2892e-05 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802902+0000", 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023515600000000001 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802908+0000", 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6890000000000001e-06 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806541+0000", 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036331800000000002 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806547+0000", 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4040000000000003e-06 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33323 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804569+0000", 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33828515599999998, 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34567882799999999, 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33323 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804569+0000", 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804569+0000", 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804570+0000", 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6600000000000002e-07 2024-05-06T00:28:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804572+0000", 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4739999999999999e-06 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804579+0000", 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7059999999999998e-06 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804585+0000", 2024-05-06T00:28:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6640000000000003e-06 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804818+0000", 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023383099999999999 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804823+0000", 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6410000000000001e-06 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807777+0000", 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029535009999999999 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807782+0000", 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1909999999999999e-06 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33386 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805428+0000", 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33742676500000002, 2024-05-06T00:28:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34490881200000001, 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33386 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805428+0000", 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805428+0000", 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805428+0000", 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3700000000000001e-07 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805430+0000", 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.916e-06 2024-05-06T00:28:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805435+0000", 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0810000000000001e-06 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805439+0000", 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0389999999999998e-06 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808733+0000", 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003294249 2024-05-06T00:28:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808739+0000", 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7130000000000002e-06 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33463 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809875+0000", 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33297901299999999, 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34052859200000002, 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33463 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809875+0000", 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809875+0000", 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809876+0000", 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2800000000000003e-07 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809878+0000", 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.26e-06 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809884+0000", 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6319999999999996e-06 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809888+0000", 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4100000000000001e-06 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812205+0000", 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023165970000000001 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812210+0000", 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33591 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810521+0000", 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33233347899999999, 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33996278499999999, 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33591 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810521+0000", 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810521+0000", 2024-05-06T00:28:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810521+0000", 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7800000000000002e-07 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810523+0000", 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0729999999999999e-06 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810535+0000", 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2089999999999999e-05 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810540+0000", 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0159999999999999e-06 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813234+0000", 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026938259999999999 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813240+0000", 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6230000000000002e-06 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33920 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811221+0000", 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33163357199999999, 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.339331781, 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33920 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811221+0000", 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811221+0000", 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811221+0000", 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811223+0000", 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0829999999999998e-06 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811229+0000", 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9510000000000003e-06 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811234+0000", 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6879999999999998e-06 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814536+0000", 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00330226 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814541+0000", 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2719999999999997e-06 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33968 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811984+0000", 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.330870726, 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.338637036, 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33968 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811984+0000", 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811984+0000", 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811984+0000", 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811986+0000", 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.024e-06 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811992+0000", 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4700000000000001e-06 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811996+0000", 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0670000000000002e-06 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816682+0000", 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046869119999999997 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816689+0000", 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7970000000000003e-06 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11611 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802061+0000", 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34079340400000002, 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34862775600000001, 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11611 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802061+0000", 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802061+0000", 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802061+0000", 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2599999999999998e-07 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802065+0000", 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1109999999999999e-06 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802071+0000", 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802077+0000", 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3179999999999997e-06 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802908+0000", 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000831824 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802913+0000", 2024-05-06T00:28:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9429999999999999e-06 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805454+0000", 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025405879999999999 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805461+0000", 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0659999999999998e-06 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11638 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802658+0000", 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34019683099999998, 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34811073300000001, 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11638 2024-05-06T00:28:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802658+0000", 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802658+0000", 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802658+0000", 2024-05-06T00:28:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.53e-07 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802661+0000", 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9900000000000002e-06 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802668+0000", 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0570000000000004e-06 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802673+0000", 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1159999999999998e-06 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803659+0000", 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098614899999999997 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803664+0000", 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0470000000000001e-06 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806213+0000", 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025489879999999999 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806219+0000", 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9159999999999998e-06 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33326 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804593+0000", 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33826175400000003, 2024-05-06T00:28:05.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34626978400000002, 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33326 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804593+0000", 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804593+0000", 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804593+0000", 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804608+0000", 2024-05-06T00:28:05.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5277999999999998e-05 2024-05-06T00:28:05.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804615+0000", 2024-05-06T00:28:05.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0380000000000002e-06 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804621+0000", 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-06 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804846+0000", 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022498200000000001 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804851+0000", 2024-05-06T00:28:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6809999999999997e-06 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807806+0000", 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029554910000000002 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807812+0000", 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6710000000000004e-06 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33388 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805444+0000", 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337410512, 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34550471300000002, 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33388 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805444+0000", 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805444+0000", 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805444+0000", 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2099999999999998e-07 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805446+0000", 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6470000000000001e-06 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805451+0000", 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9679999999999997e-06 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805456+0000", 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.809e-06 2024-05-06T00:28:05.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808764+0000", 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003308456 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808770+0000", 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.891e-06 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33465 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809894+0000", 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33296087299999999, 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34112664700000001, 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33465 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809894+0000", 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809894+0000", 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809894+0000", 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4299999999999999e-07 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809896+0000", 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9599999999999999e-06 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809901+0000", 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0239999999999997e-06 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809905+0000", 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:05.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812238+0000", 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023335130000000002 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812244+0000", 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4689999999999996e-06 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33593 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810547+0000", 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33230706799999998, 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34057950599999998, 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33593 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810547+0000", 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810547+0000", 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810548+0000", 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1600000000000002e-07 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810556+0000", 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5329999999999994e-06 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810561+0000", 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810566+0000", 2024-05-06T00:28:05.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8709999999999996e-06 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813266+0000", 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027000890000000001 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813272+0000", 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7309999999999998e-06 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33921 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811240+0000", 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33161423000000001, 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.339972881, 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33921 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811240+0000", 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811240+0000", 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811241+0000", 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1899999999999998e-07 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811243+0000", 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1399999999999998e-06 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811250+0000", 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2110000000000001e-06 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811255+0000", 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1100000000000002e-06 2024-05-06T00:28:05.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814772+0000", 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003516618 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814781+0000", 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3840000000000003e-06 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.611 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33970 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812008+0000", 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33084624000000001, 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33927347299999999, 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33970 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812008+0000", 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812008+0000", 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812008+0000", 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812011+0000", 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1270000000000001e-06 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812016+0000", 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7409999999999998e-06 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812020+0000", 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1289999999999998e-06 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816716+0000", 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046955 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816723+0000", 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2450000000000001e-06 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11612 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802084+0000", 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34077069599999998, 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34930135099999998, 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11612 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802084+0000", 2024-05-06T00:28:05.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802084+0000", 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802084+0000", 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5900000000000003e-07 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802087+0000", 2024-05-06T00:28:05.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.965e-06 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802094+0000", 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6200000000000001e-06 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802099+0000", 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.378e-06 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802937+0000", 2024-05-06T00:28:05.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00083771400000000004 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802942+0000", 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0139999999999998e-06 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805482+0000", 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025400959999999999 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805487+0000", 2024-05-06T00:28:05.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2329999999999997e-06 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11639 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802681+0000", 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34017340200000001, 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.348784025, 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11639 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802681+0000", 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802681+0000", 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802681+0000", 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8200000000000001e-07 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802684+0000", 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0070000000000002e-06 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802691+0000", 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1809999999999998e-06 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802695+0000", 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5190000000000003e-06 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803687+0000", 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00099210100000000001 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803692+0000", 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6240000000000001e-06 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806241+0000", 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002549571 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806247+0000", 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.981e-06 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33328 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804629+0000", 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33822573299999997, 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34691597800000001, 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33328 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804629+0000", 2024-05-06T00:28:05.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804629+0000", 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804629+0000", 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.58e-07 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804631+0000", 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804637+0000", 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6389999999999997e-06 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804642+0000", 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9219999999999996e-06 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804880+0000", 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023856800000000001 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804885+0000", 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5809999999999998e-06 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807836+0000", 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029511149999999998 2024-05-06T00:28:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807842+0000", 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7150000000000003e-06 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33404 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805675+0000", 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33717980199999997, 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34594926399999998, 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33404 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805675+0000", 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805675+0000", 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805675+0000", 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805679+0000", 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.45e-06 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805689+0000", 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9029999999999999e-06 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805695+0000", 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999997e-06 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808795+0000", 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030997279999999999 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808800+0000", 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4489999999999998e-06 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33468 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809911+0000", 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.332943826, 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34178060799999999, 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33468 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809911+0000", 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809911+0000", 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809911+0000", 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809913+0000", 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4140000000000001e-06 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809918+0000", 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0579999999999997e-06 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809923+0000", 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2989999999999998e-06 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812271+0000", 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002348185 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812277+0000", 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6339999999999998e-06 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33595 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810573+0000", 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33228187999999997, 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34118605600000002, 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33595 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810573+0000", 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810573+0000", 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810573+0000", 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6100000000000001e-07 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810575+0000", 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0499999999999999e-06 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810582+0000", 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5590000000000002e-06 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810586+0000", 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7210000000000002e-06 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813297+0000", 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027107110000000002 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813303+0000", 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9710000000000001e-06 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33922 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811261+0000", 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.331593103, 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34056447899999998, 2024-05-06T00:28:05.641 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33922 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811261+0000", 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811261+0000", 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811262+0000", 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999998e-07 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811264+0000", 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.142e-06 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811271+0000", 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.883e-06 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811276+0000", 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.502e-06 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814829+0000", 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035533000000000001 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814837+0000", 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5700000000000004e-06 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33972 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812026+0000", 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33082806199999998, 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.339866528, 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33972 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812026+0000", 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812026+0000", 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812027+0000", 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.39e-07 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812029+0000", 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.86e-06 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812034+0000", 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3070000000000002e-06 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812038+0000", 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9620000000000004e-06 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816752+0000", 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047142099999999999 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816760+0000", 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1159999999999999e-06 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33176 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802085+0000", 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34076925600000002, 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34987462699999999, 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.649 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33176 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802085+0000", 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802085+0000", 2024-05-06T00:28:05.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802086+0000", 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6899999999999996e-07 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802097+0000", 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1364e-05 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802106+0000", 2024-05-06T00:28:05.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8850000000000005e-06 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802113+0000", 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4350000000000002e-06 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802195+0000", 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2114999999999996e-05 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802201+0000", 2024-05-06T00:28:05.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806428+0000", 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042275100000000003 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806434+0000", 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6269999999999997e-06 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11641 3.10 3.f419e410 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802704+0000", 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34015083000000002, 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34933619799999999, 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11641 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802704+0000", 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802704+0000", 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802704+0000", 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5700000000000002e-07 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802707+0000", 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.571e-06 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802714+0000", 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7769999999999997e-06 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802719+0000", 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9629999999999997e-06 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803715+0000", 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00099687800000000009 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803720+0000", 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.989e-06 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806269+0000", 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025483770000000001 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806275+0000", 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.72e-06 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33330 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804727+0000", 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33812719299999999, 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34739204800000001, 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33330 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804727+0000", 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804727+0000", 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804728+0000", 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2099999999999997e-07 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804734+0000", 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8969999999999996e-06 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804742+0000", 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7730000000000005e-06 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804748+0000", 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9959999999999999e-06 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804912+0000", 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016394399999999999 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804918+0000", 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.378e-06 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807866+0000", 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029482060000000001 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807872+0000", 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.784e-06 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33406 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805705+0000", 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33714948300000003, 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34649322799999999, 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33406 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805705+0000", 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805705+0000", 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805705+0000", 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6600000000000002e-07 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805711+0000", 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3240000000000002e-06 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805718+0000", 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3100000000000003e-06 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805723+0000", 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808824+0000", 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031013970000000001 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808830+0000", 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2950000000000001e-06 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33470 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809929+0000", 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.332925376, 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34234753899999998, 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33470 2024-05-06T00:28:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809929+0000", 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809929+0000", 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809930+0000", 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2799999999999996e-07 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809936+0000", 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2230000000000004e-06 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809942+0000", 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7479999999999999e-06 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809946+0000", 2024-05-06T00:28:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999996e-06 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812304+0000", 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023576040000000001 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812315+0000", 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1068e-05 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33597 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810593+0000", 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33226164000000002, 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34175201599999999, 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33597 2024-05-06T00:28:05.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810593+0000", 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810593+0000", 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810593+0000", 2024-05-06T00:28:05.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6300000000000005e-07 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810602+0000", 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3580000000000003e-06 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810609+0000", 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2819999999999999e-06 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810614+0000", 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-06 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813330+0000", 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002715463 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813335+0000", 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7980000000000002e-06 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33923 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811283+0000", 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33157160200000002, 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34112943699999998, 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33923 2024-05-06T00:28:05.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811283+0000", 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811283+0000", 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811283+0000", 2024-05-06T00:28:05.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7399999999999999e-07 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811292+0000", 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8610000000000004e-06 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811299+0000", 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9439999999999999e-06 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811304+0000", 2024-05-06T00:28:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8319999999999996e-06 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814865+0000", 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035610379999999999 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814872+0000", 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0230000000000004e-06 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33974 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812043+0000", 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33081147500000002, 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34047925899999998, 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33974 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812043+0000", 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812043+0000", 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812043+0000", 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812045+0000", 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6530000000000001e-06 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812050+0000", 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9130000000000002e-06 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812054+0000", 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8600000000000003e-06 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816789+0000", 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047357559999999998 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816798+0000", 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2959999999999999e-06 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11613 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802106+0000", 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34074871299999998, 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.350542627, 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11613 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802106+0000", 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802106+0000", 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802106+0000", 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6100000000000001e-07 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802109+0000", 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0910000000000001e-06 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802116+0000", 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4160000000000001e-06 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802121+0000", 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6520000000000003e-06 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802965+0000", 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00084323999999999998 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802970+0000", 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9439999999999996e-06 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805508+0000", 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002538252 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805513+0000", 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5060000000000003e-06 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33185 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802719+0000", 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34013507900000001, 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35001168700000002, 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33185 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802719+0000", 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802719+0000", 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802720+0000", 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6099999999999999e-07 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802723+0000", 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2619999999999999e-06 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802743+0000", 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9298999999999999e-05 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802749+0000", 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9580000000000004e-06 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802996+0000", 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024766000000000002 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803002+0000", 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.699e-06 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806569+0000", 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035671840000000002 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806574+0000", 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.691 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33332 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804759+0000", 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33809537499999998, 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34805739299999999, 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33332 2024-05-06T00:28:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804759+0000", 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804759+0000", 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804759+0000", 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8299999999999997e-07 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804762+0000", 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5969999999999999e-06 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804769+0000", 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.759e-06 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804774+0000", 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5210000000000002e-06 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804938+0000", 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163843 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804943+0000", 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.899e-06 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807896+0000", 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029524 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807901+0000", 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9100000000000002e-06 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.696 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33408 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805730+0000", 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33712439799999999, 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34716625000000001, 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33408 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805730+0000", 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805730+0000", 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805730+0000", 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805732+0000", 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9750000000000001e-06 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805737+0000", 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8040000000000001e-06 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805741+0000", 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9380000000000002e-06 2024-05-06T00:28:05.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808855+0000", 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031138030000000001 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808860+0000", 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2109999999999997e-06 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.700 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33472 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809953+0000", 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33290186599999999, 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34301125700000001, 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33472 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809953+0000", 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809953+0000", 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809953+0000", 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6100000000000002e-07 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809955+0000", 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3089999999999998e-06 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809961+0000", 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6180000000000003e-06 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809966+0000", 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9629999999999997e-06 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812343+0000", 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002377353 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812349+0000", 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8649999999999998e-06 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33599 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.704 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810621+0000", 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33223335199999998, 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34241193800000003, 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33599 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810621+0000", 2024-05-06T00:28:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810621+0000", 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810621+0000", 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4200000000000002e-07 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810624+0000", 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2290000000000002e-06 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810629+0000", 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6239999999999999e-06 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810634+0000", 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8470000000000003e-06 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813362+0000", 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027276010000000001 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813368+0000", 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8289999999999996e-06 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33924 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811310+0000", 2024-05-06T00:28:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33154401700000002, 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34179055800000002, 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33924 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811310+0000", 2024-05-06T00:28:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811310+0000", 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811311+0000", 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8500000000000002e-07 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811313+0000", 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1409999999999999e-06 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811319+0000", 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6699999999999999e-06 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811323+0000", 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8590000000000004e-06 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814899+0000", 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035753400000000002 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814905+0000", 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6810000000000001e-06 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34102 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812069+0000", 2024-05-06T00:28:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33078518200000001, 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34109959499999998, 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34102 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812069+0000", 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812069+0000", 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812070+0000", 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3799999999999998e-07 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812078+0000", 2024-05-06T00:28:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8240000000000006e-06 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812084+0000", 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9959999999999999e-06 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812089+0000", 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3980000000000001e-06 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816826+0000", 2024-05-06T00:28:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047373199999999997 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816839+0000", 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3131e-05 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11614 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802128+0000", 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34072655600000001, 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.351120139, 2024-05-06T00:28:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11614 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802128+0000", 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802128+0000", 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802128+0000", 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802131+0000", 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7310000000000002e-06 2024-05-06T00:28:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802137+0000", 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0469999999999999e-06 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802142+0000", 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3990000000000003e-06 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802994+0000", 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00085143399999999998 2024-05-06T00:28:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802998+0000", 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3699999999999997e-06 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805537+0000", 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025385450000000001 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805543+0000", 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6730000000000003e-06 2024-05-06T00:28:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33186 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802775+0000", 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34007921400000002, 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35057845700000001, 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33186 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802775+0000", 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802775+0000", 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802776+0000", 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6000000000000004e-07 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802779+0000", 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9349999999999999e-06 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802788+0000", 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2299999999999997e-06 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802794+0000", 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8089999999999998e-06 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803032+0000", 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000238218 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803037+0000", 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6229999999999996e-06 2024-05-06T00:28:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806597+0000", 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035606710000000001 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806603+0000", 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.66e-06 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33334 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804782+0000", 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33807257899999998, 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34869773700000001, 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33334 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804782+0000", 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804782+0000", 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804782+0000", 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804784+0000", 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2290000000000002e-06 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804793+0000", 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1289999999999996e-06 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804798+0000", 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0799999999999996e-06 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804970+0000", 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017190200000000001 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804974+0000", 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3320000000000002e-06 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807934+0000", 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029597790000000001 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807940+0000", 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2539999999999998e-06 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33411 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805748+0000", 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33710625599999999, 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.347813233, 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33411 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805748+0000", 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805748+0000", 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805748+0000", 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8299999999999998e-07 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805765+0000", 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6577000000000001e-05 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805772+0000", 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3370000000000002e-06 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805779+0000", 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808884+0000", 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003105226 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808889+0000", 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3079999999999998e-06 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33474 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809971+0000", 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33288293299999999, 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34365737099999999, 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33474 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809971+0000", 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809971+0000", 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809972+0000", 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.65e-07 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809974+0000", 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9010000000000001e-06 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809980+0000", 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9399999999999999e-06 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809985+0000", 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9869999999999999e-06 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812380+0000", 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023949779999999999 2024-05-06T00:28:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812385+0000", 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8590000000000001e-06 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33601 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810641+0000", 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33221383799999998, 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.343056262, 2024-05-06T00:28:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33601 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810641+0000", 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810641+0000", 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810641+0000", 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810643+0000", 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0710000000000002e-06 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810649+0000", 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0990000000000004e-06 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810654+0000", 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7650000000000001e-06 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813396+0000", 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027421759999999998 2024-05-06T00:28:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813402+0000", 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0410000000000003e-06 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33925 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811330+0000", 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33152458899999998, 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34243459500000001, 2024-05-06T00:28:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33925 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811330+0000", 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811330+0000", 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811330+0000", 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4700000000000002e-07 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811337+0000", 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999997e-06 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811341+0000", 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5809999999999998e-06 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811346+0000", 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9189999999999998e-06 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814934+0000", 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003588089 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814941+0000", 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7710000000000001e-06 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34103 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812106+0000", 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33074834600000003, 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34172555999999998, 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34103 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812106+0000", 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812106+0000", 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812106+0000", 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4200000000000002e-07 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812109+0000", 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1129999999999999e-06 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812115+0000", 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0909999999999998e-06 2024-05-06T00:28:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812119+0000", 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5510000000000001e-06 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816868+0000", 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047491540000000002 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816876+0000", 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3119999999999996e-06 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33178 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802139+0000", 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34071515899999999, 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35176007799999998, 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33178 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802139+0000", 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802139+0000", 2024-05-06T00:28:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802140+0000", 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7000000000000004e-07 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802143+0000", 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2059999999999999e-06 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802152+0000", 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4139999999999994e-06 2024-05-06T00:28:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802166+0000", 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4389999999999999e-05 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802228+0000", 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1653000000000001e-05 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802233+0000", 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2179999999999998e-06 2024-05-06T00:28:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806457+0000", 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042240339999999998 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806462+0000", 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5040000000000002e-06 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33188 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802937+0000", 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33991785200000002, 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35104247500000002, 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33188 2024-05-06T00:28:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802937+0000", 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802937+0000", 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802937+0000", 2024-05-06T00:28:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6000000000000003e-07 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802945+0000", 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9170000000000006e-06 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802953+0000", 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9889999999999992e-06 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802959+0000", 2024-05-06T00:28:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1290000000000001e-06 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803324+0000", 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036476899999999998 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803330+0000", 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8420000000000001e-06 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806626+0000", 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032962600000000001 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806632+0000", 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4589999999999997e-06 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33336 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804814+0000", 2024-05-06T00:28:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33804004500000001, 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34924422500000002, 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33336 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804814+0000", 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804814+0000", 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804815+0000", 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7300000000000001e-07 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804818+0000", 2024-05-06T00:28:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.706e-06 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804825+0000", 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2640000000000002e-06 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804830+0000", 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1919999999999996e-06 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805439+0000", 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000609203 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805448+0000", 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0140000000000004e-06 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807964+0000", 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025162119999999999 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807970+0000", 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8629999999999996e-06 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33413 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805786+0000", 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337068425, 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34835176699999998, 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33413 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805786+0000", 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805786+0000", 2024-05-06T00:28:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805786+0000", 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.08e-07 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805789+0000", 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1550000000000001e-06 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805794+0000", 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3550000000000004e-06 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805799+0000", 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9620000000000001e-06 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808913+0000", 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031141989999999998 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808918+0000", 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.169e-06 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33475 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.809991+0000", 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33286360399999998, 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34421425900000002, 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33475 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809991+0000", 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809991+0000", 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809991+0000", 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.77e-07 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809993+0000", 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1009999999999999e-06 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809999+0000", 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7100000000000004e-06 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810003+0000", 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3680000000000004e-06 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812413+0000", 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024092580000000001 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812418+0000", 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8150000000000002e-06 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33603 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810660+0000", 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.332194249, 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34361190200000002, 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33603 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810660+0000", 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810660+0000", 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810661+0000", 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810663+0000", 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0320000000000002e-06 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810669+0000", 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1539999999999999e-06 2024-05-06T00:28:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810674+0000", 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8640000000000004e-06 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813428+0000", 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027541340000000001 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813434+0000", 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.93e-06 2024-05-06T00:28:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33926 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811353+0000", 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33150181000000001, 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34298671200000003, 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33926 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811353+0000", 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811353+0000", 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811353+0000", 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811355+0000", 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0209999999999998e-06 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811363+0000", 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7230000000000004e-06 2024-05-06T00:28:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811368+0000", 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1240000000000004e-06 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814968+0000", 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003599802 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.814975+0000", 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0929999999999997e-06 2024-05-06T00:28:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34104 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812135+0000", 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33071982100000002, 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34227182099999998, 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34104 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812135+0000", 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812135+0000", 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812135+0000", 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3799999999999998e-07 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812137+0000", 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812143+0000", 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8610000000000003e-06 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812148+0000", 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5620000000000003e-06 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816905+0000", 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047565309999999996 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816912+0000", 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4499999999999998e-06 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11615 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802151+0000", 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340703698, 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35232319000000001, 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11615 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802151+0000", 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802151+0000", 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802151+0000", 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7299999999999995e-07 2024-05-06T00:28:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802154+0000", 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0699999999999998e-06 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802161+0000", 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8229999999999997e-06 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802167+0000", 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3759999999999999e-06 2024-05-06T00:28:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803021+0000", 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00085392599999999997 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803025+0000", 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.78e-06 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805563+0000", 2024-05-06T00:28:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025378940000000002 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805569+0000", 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9449999999999998e-06 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33189 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803007+0000", 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.339847819, 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35161490000000001, 2024-05-06T00:28:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33189 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803007+0000", 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803007+0000", 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803007+0000", 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8899999999999999e-07 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803010+0000", 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7420000000000002e-06 2024-05-06T00:28:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803019+0000", 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1970000000000002e-06 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803031+0000", 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2058999999999999e-05 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803361+0000", 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000329711 2024-05-06T00:28:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803366+0000", 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.023e-06 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806655+0000", 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032890699999999998 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806661+0000", 2024-05-06T00:28:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5439999999999998e-06 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33338 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804839+0000", 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33801575, 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34989724300000002, 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33338 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804839+0000", 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804839+0000", 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804839+0000", 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9999999999999998e-07 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804842+0000", 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.587e-06 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804848+0000", 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0599999999999996e-06 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804853+0000", 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8679999999999998e-06 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805487+0000", 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063411299999999995 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805492+0000", 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1569999999999999e-06 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807994+0000", 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025021800000000001 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808000+0000", 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33414 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805806+0000", 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337048866, 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34902513299999999, 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33414 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805806+0000", 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805806+0000", 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805806+0000", 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805808+0000", 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0439999999999998e-06 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805814+0000", 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2740000000000004e-06 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805820+0000", 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7760000000000003e-06 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808943+0000", 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031227160000000002 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808948+0000", 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2700000000000004e-06 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33477 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810009+0000", 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33284583600000001, 2024-05-06T00:28:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34489140899999998, 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33477 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810009+0000", 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810009+0000", 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810009+0000", 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1699999999999999e-07 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810011+0000", 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1390000000000002e-06 2024-05-06T00:28:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810016+0000", 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3959999999999997e-06 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810021+0000", 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5299999999999998e-06 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812447+0000", 2024-05-06T00:28:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024257329999999998 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812453+0000", 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9479999999999996e-06 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33605 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810680+0000", 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33217475499999999, 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34428917399999998, 2024-05-06T00:28:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33605 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810680+0000", 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810680+0000", 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810680+0000", 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3000000000000001e-07 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810682+0000", 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0559999999999999e-06 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810689+0000", 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2980000000000003e-06 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810696+0000", 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0909999999999998e-06 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813456+0000", 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027600429999999998 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813462+0000", 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9020000000000002e-06 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33927 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811375+0000", 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.331479678, 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34366311300000002, 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33927 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811375+0000", 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811375+0000", 2024-05-06T00:28:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811375+0000", 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8099999999999998e-07 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811377+0000", 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811383+0000", 2024-05-06T00:28:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2099999999999998e-06 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811388+0000", 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8319999999999996e-06 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815008+0000", 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036196179999999998 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815015+0000", 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4189999999999996e-06 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34105 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812169+0000", 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33068489299999998, 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34293652800000002, 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34105 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812169+0000", 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812169+0000", 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812170+0000", 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.27e-07 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812172+0000", 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0600000000000002e-06 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812178+0000", 2024-05-06T00:28:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0039999999999996e-06 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812183+0000", 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4020000000000001e-06 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816939+0000", 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047561799999999996 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816946+0000", 2024-05-06T00:28:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7669999999999998e-06 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11616 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802174+0000", 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34068074700000001, 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35300026099999998, 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11616 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802174+0000", 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802174+0000", 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802174+0000", 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802177+0000", 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802184+0000", 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.652e-06 2024-05-06T00:28:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802189+0000", 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4149999999999998e-06 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803048+0000", 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00085853000000000004 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803052+0000", 2024-05-06T00:28:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5240000000000002e-06 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805591+0000", 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00253879 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805597+0000", 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6160000000000001e-06 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33190 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803053+0000", 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.339801509, 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35221044899999998, 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33190 2024-05-06T00:28:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803053+0000", 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803053+0000", 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803053+0000", 2024-05-06T00:28:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803056+0000", 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7539999999999998e-06 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803066+0000", 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3379999999999994e-06 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803077+0000", 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1527e-05 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803401+0000", 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032424599999999999 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803414+0000", 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2877000000000001e-05 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806684+0000", 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032699090000000001 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806689+0000", 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4210000000000003e-06 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33340 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804876+0000", 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33797825999999997, 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35047887100000003, 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33340 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804876+0000", 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804876+0000", 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804877+0000", 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6600000000000002e-07 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804879+0000", 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8270000000000002e-06 2024-05-06T00:28:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804887+0000", 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2300000000000002e-06 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804892+0000", 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9429999999999999e-06 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808026+0000", 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031346809999999998 2024-05-06T00:28:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808032+0000", 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2830000000000001e-06 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33416 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805827+0000", 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33702769700000001, 2024-05-06T00:28:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.349597086, 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33416 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805827+0000", 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805827+0000", 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805827+0000", 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5199999999999998e-07 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805829+0000", 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2529999999999999e-06 2024-05-06T00:28:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805835+0000", 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0050000000000001e-06 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805841+0000", 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3310000000000003e-06 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808972+0000", 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031315819999999999 2024-05-06T00:28:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808977+0000", 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0130000000000001e-06 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33479 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810027+0000", 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33282725000000002, 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34550026700000003, 2024-05-06T00:28:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33479 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810027+0000", 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810027+0000", 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810028+0000", 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810030+0000", 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0509999999999999e-06 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810037+0000", 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3209999999999999e-06 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810042+0000", 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9099999999999996e-06 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812478+0000", 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024364930000000001 2024-05-06T00:28:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812484+0000", 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5790000000000003e-06 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33607 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810709+0000", 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33214570599999999, 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.344906565, 2024-05-06T00:28:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33607 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810709+0000", 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810709+0000", 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810709+0000", 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810711+0000", 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1469999999999999e-06 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810718+0000", 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0520000000000004e-06 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810723+0000", 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9679999999999997e-06 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813486+0000", 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027623729999999998 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813491+0000", 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2209999999999996e-06 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33928 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811395+0000", 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33145960699999999, 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34428950800000002, 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33928 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811395+0000", 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811395+0000", 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811395+0000", 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8500000000000002e-07 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811397+0000", 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0499999999999999e-06 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811403+0000", 2024-05-06T00:28:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9880000000000001e-06 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811408+0000", 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815043+0000", 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003634871 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815050+0000", 2024-05-06T00:28:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.024e-06 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34106 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812202+0000", 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33065225500000001, 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34354963799999999, 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34106 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812202+0000", 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812202+0000", 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812202+0000", 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6199999999999999e-07 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812205+0000", 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.357e-06 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812212+0000", 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.703e-06 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812217+0000", 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4809999999999997e-06 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816973+0000", 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047559070000000002 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816980+0000", 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7850000000000003e-06 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11617 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802196+0000", 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34065855099999998, 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35362332200000002, 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11617 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802196+0000", 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802196+0000", 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802197+0000", 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6000000000000003e-07 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802199+0000", 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.886e-06 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802206+0000", 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4819999999999999e-06 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802212+0000", 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-06 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803074+0000", 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086206200000000003 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803078+0000", 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805618+0000", 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025394979999999998 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805623+0000", 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3959999999999997e-06 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33192 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803086+0000", 2024-05-06T00:28:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33976872899999999, 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.352812976, 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33192 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803086+0000", 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803086+0000", 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803086+0000", 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8299999999999997e-07 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803107+0000", 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0735999999999999e-05 2024-05-06T00:28:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803115+0000", 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.174e-06 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803121+0000", 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7760000000000003e-06 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803473+0000", 2024-05-06T00:28:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035245699999999998 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803479+0000", 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4029999999999998e-06 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806712+0000", 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032335490000000001 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806717+0000", 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6750000000000001e-06 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33341 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804909+0000", 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33794523700000001, 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35106868299999999, 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33341 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804909+0000", 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804909+0000", 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804910+0000", 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4700000000000002e-07 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804912+0000", 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6479999999999999e-06 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804919+0000", 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.725e-06 2024-05-06T00:28:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804925+0000", 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7570000000000001e-06 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808055+0000", 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003130412 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808060+0000", 2024-05-06T00:28:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2349999999999999e-06 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33418 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805847+0000", 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337007257, 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35019760999999999, 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33418 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805847+0000", 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805847+0000", 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805847+0000", 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805850+0000", 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0530000000000001e-06 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805857+0000", 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6399999999999997e-06 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805862+0000", 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2050000000000001e-06 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809001+0000", 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031387260000000001 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809006+0000", 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3349999999999997e-06 2024-05-06T00:28:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33481 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810048+0000", 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.332806826, 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34606452100000001, 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33481 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810048+0000", 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810048+0000", 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810048+0000", 2024-05-06T00:28:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4799999999999999e-07 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810065+0000", 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6843000000000001e-05 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810071+0000", 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5699999999999998e-06 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810078+0000", 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5379999999999999e-06 2024-05-06T00:28:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812510+0000", 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024325739999999998 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812516+0000", 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7679999999999997e-06 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33609 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810730+0000", 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33212434800000001, 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34544939299999999, 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33609 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810730+0000", 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810730+0000", 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810730+0000", 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5199999999999998e-07 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810733+0000", 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.199e-06 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810738+0000", 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8089999999999998e-06 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810743+0000", 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813516+0000", 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027730680000000001 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813522+0000", 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33929 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811415+0000", 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33143982999999999, 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34483241799999997, 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33929 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811415+0000", 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811415+0000", 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811415+0000", 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8200000000000001e-07 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811417+0000", 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.215e-06 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811423+0000", 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811428+0000", 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815078+0000", 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036498329999999999 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815085+0000", 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1099999999999997e-06 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34107 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812235+0000", 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33061966100000001, 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34407965800000001, 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34107 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812235+0000", 2024-05-06T00:28:05.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812235+0000", 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812235+0000", 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2100000000000002e-07 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812237+0000", 2024-05-06T00:28:05.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2759999999999999e-06 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812244+0000", 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7870000000000004e-06 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812249+0000", 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9819999999999999e-06 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817004+0000", 2024-05-06T00:28:05.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047544889999999998 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817011+0000", 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8800000000000002e-06 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33179 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802191+0000", 2024-05-06T00:28:05.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340663204, 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35420094000000002, 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33179 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802191+0000", 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802191+0000", 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802192+0000", 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3099999999999997e-07 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802195+0000", 2024-05-06T00:28:05.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802220+0000", 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4601999999999999e-05 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802235+0000", 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5651999999999999e-05 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802659+0000", 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042358400000000002 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802665+0000", 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806485+0000", 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038199520000000002 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806491+0000", 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4e-06 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33193 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803136+0000", 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33971818599999998, 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.353335915, 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33193 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803136+0000", 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803136+0000", 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803137+0000", 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803139+0000", 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3810000000000002e-06 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803146+0000", 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5520000000000001e-06 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803151+0000", 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3419999999999998e-06 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803513+0000", 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036211300000000001 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803518+0000", 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.865e-06 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806738+0000", 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032196400000000002 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806743+0000", 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3229999999999997e-06 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33343 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804933+0000", 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33792127700000002, 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35161831100000002, 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33343 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804933+0000", 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804933+0000", 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804934+0000", 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.96e-07 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804936+0000", 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.266e-06 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804942+0000", 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9209999999999997e-06 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804947+0000", 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4909999999999996e-06 2024-05-06T00:28:05.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808085+0000", 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003137562 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808090+0000", 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33420 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805869+0000", 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.336985113, 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35074904600000001, 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33420 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805869+0000", 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805869+0000", 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805870+0000", 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8500000000000002e-07 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805872+0000", 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3149999999999999e-06 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805878+0000", 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4400000000000002e-06 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805885+0000", 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.63e-06 2024-05-06T00:28:05.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809030+0000", 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00314525 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809036+0000", 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3319999999999999e-06 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33482 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810086+0000", 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33276856500000002, 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34661125799999998, 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33482 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810086+0000", 2024-05-06T00:28:05.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810086+0000", 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810086+0000", 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.08e-07 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810089+0000", 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.542e-06 2024-05-06T00:28:05.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810095+0000", 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1290000000000001e-06 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810100+0000", 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1499999999999998e-06 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812541+0000", 2024-05-06T00:28:05.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002441005 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7139999999999998e-06 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33611 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810750+0000", 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33210463000000001, 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34601489400000002, 2024-05-06T00:28:05.903 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33611 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810750+0000", 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810750+0000", 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810750+0000", 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4400000000000001e-07 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810752+0000", 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0679999999999999e-06 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810759+0000", 2024-05-06T00:28:05.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9020000000000002e-06 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810764+0000", 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8899999999999998e-06 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813544+0000", 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027803720000000001 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813550+0000", 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6790000000000002e-06 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33930 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811434+0000", 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33142017800000001, 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34546511400000002, 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.907 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33930 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811434+0000", 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811434+0000", 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811435+0000", 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6600000000000002e-07 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811437+0000", 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1289999999999999e-06 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811444+0000", 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.951e-06 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811449+0000", 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0839999999999999e-06 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815113+0000", 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036639839999999999 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815120+0000", 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3760000000000002e-06 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34109 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812271+0000", 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33058333600000001, 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34472745799999999, 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34109 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.911 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812271+0000", 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812271+0000", 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812272+0000", 2024-05-06T00:28:05.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4499999999999997e-07 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812274+0000", 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2680000000000001e-06 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812280+0000", 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.618e-06 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812285+0000", 2024-05-06T00:28:05.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9729999999999997e-06 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817419+0000", 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051338119999999998 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817429+0000", 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9159999999999996e-06 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11618 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802219+0000", 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34063567700000003, 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35484937300000002, 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11618 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802219+0000", 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802219+0000", 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802219+0000", 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-05-06T00:28:05.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802222+0000", 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802229+0000", 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5390000000000004e-06 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802234+0000", 2024-05-06T00:28:05.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5929999999999996e-06 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803099+0000", 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086469200000000002 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803103+0000", 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6779999999999999e-06 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805645+0000", 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002541855 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805651+0000", 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2360000000000004e-06 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33195 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803160+0000", 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.339694406, 2024-05-06T00:28:05.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.353990095, 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33195 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803160+0000", 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803160+0000", 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803160+0000", 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.96e-07 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803162+0000", 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1179999999999999e-06 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803177+0000", 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4336e-05 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803182+0000", 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5679999999999999e-06 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803545+0000", 2024-05-06T00:28:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000362312 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803549+0000", 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6550000000000003e-06 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806766+0000", 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032166930000000001 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806772+0000", 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5300000000000004e-06 2024-05-06T00:28:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33345 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804966+0000", 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33788855299999998, 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35226363900000002, 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33345 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804966+0000", 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804966+0000", 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804966+0000", 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3000000000000001e-07 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804969+0000", 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8890000000000002e-06 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804977+0000", 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1990000000000006e-06 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804982+0000", 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0019999999999997e-06 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808114+0000", 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003132003 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808120+0000", 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33422 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805891+0000", 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33696299499999999, 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35140634599999998, 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33422 2024-05-06T00:28:05.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805891+0000", 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805891+0000", 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805892+0000", 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:05.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805894+0000", 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1900000000000002e-06 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805900+0000", 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805905+0000", 2024-05-06T00:28:05.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9779999999999996e-06 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809060+0000", 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003154741 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809065+0000", 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.198e-06 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33484 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810106+0000", 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33274810100000002, 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.347258909, 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33484 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810106+0000", 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810106+0000", 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810107+0000", 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4700000000000002e-07 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810109+0000", 2024-05-06T00:28:05.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.063e-06 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810115+0000", 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.925e-06 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810120+0000", 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9069999999999998e-06 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812576+0000", 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002456307 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9410000000000004e-06 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33613 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810770+0000", 2024-05-06T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33208417699999998, 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34666200699999999, 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33613 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810770+0000", 2024-05-06T00:28:05.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810770+0000", 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810771+0000", 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810773+0000", 2024-05-06T00:28:05.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9999999999999999e-06 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810779+0000", 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7959999999999998e-06 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810784+0000", 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9339999999999997e-06 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813575+0000", 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027903339999999998 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813580+0000", 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5090000000000001e-06 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33931 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811455+0000", 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33139903999999998, 2024-05-06T00:28:05.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.346044925, 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33931 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811455+0000", 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811455+0000", 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811456+0000", 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.96e-07 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811458+0000", 2024-05-06T00:28:05.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1720000000000001e-06 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811464+0000", 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6110000000000001e-06 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811469+0000", 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.271e-06 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815147+0000", 2024-05-06T00:28:05.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036780099999999998 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815154+0000", 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4150000000000002e-06 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34110 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812301+0000", 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33055335699999999, 2024-05-06T00:28:05.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.345266517, 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34110 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812301+0000", 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812301+0000", 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812301+0000", 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3500000000000002e-07 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812304+0000", 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.187e-06 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812322+0000", 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8570999999999999e-05 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812328+0000", 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817459+0000", 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051308489999999998 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817813+0000", 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035410200000000002 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11619 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802241+0000", 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340613781, 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35539443900000001, 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11619 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802241+0000", 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802241+0000", 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802241+0000", 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802250+0000", 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5809999999999997e-06 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802256+0000", 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802262+0000", 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6289999999999998e-06 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803125+0000", 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086323599999999997 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803130+0000", 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7650000000000001e-06 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805670+0000", 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025404989999999999 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805676+0000", 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4720000000000003e-06 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33197 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803196+0000", 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.339658707, 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35453168699999998, 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33197 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803196+0000", 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803196+0000", 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803196+0000", 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.06e-07 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803199+0000", 2024-05-06T00:28:05.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4399999999999999e-06 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803212+0000", 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3485999999999999e-05 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803218+0000", 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6779999999999997e-06 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803574+0000", 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000356335 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803579+0000", 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806795+0000", 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032157940000000001 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806800+0000", 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6860000000000003e-06 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33347 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804990+0000", 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337864373, 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.3528172, 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33347 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804990+0000", 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804990+0000", 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804990+0000", 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804993+0000", 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804999+0000", 2024-05-06T00:28:05.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2389999999999999e-06 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805004+0000", 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8849999999999998e-06 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808144+0000", 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031399430000000001 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808149+0000", 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.203e-06 2024-05-06T00:28:05.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33424 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805911+0000", 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.336943188, 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35198981400000001, 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33424 2024-05-06T00:28:05.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805911+0000", 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805911+0000", 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805912+0000", 2024-05-06T00:28:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805914+0000", 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0030000000000001e-06 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805920+0000", 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3269999999999997e-06 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805929+0000", 2024-05-06T00:28:05.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9800000000000004e-06 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809089+0000", 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031604760000000002 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809095+0000", 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7899999999999996e-06 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33485 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810125+0000", 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33272902500000001, 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34786220000000001, 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33485 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.964 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810125+0000", 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810125+0000", 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810126+0000", 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.39e-07 2024-05-06T00:28:05.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810128+0000", 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810134+0000", 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0599999999999996e-06 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810142+0000", 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0960000000000001e-06 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002466541 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812614+0000", 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7699999999999998e-06 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.967 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33616 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810791+0000", 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33206347899999999, 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34726521599999999, 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33616 2024-05-06T00:28:05.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810791+0000", 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810791+0000", 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810791+0000", 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8200000000000001e-07 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810793+0000", 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1260000000000001e-06 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810800+0000", 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4840000000000001e-06 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810813+0000", 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813605+0000", 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002792143 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813611+0000", 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6289999999999998e-06 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33932 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811475+0000", 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33137894899999998, 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34664824999999999, 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33932 2024-05-06T00:28:05.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811475+0000", 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811475+0000", 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811476+0000", 2024-05-06T00:28:05.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811478+0000", 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1560000000000002e-06 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811485+0000", 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811494+0000", 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1509999999999993e-06 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815224+0000", 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037298700000000001 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815232+0000", 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7300000000000005e-06 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34111 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812337+0000", 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33051787999999999, 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34585439000000001, 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34111 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.976 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812337+0000", 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812337+0000", 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812337+0000", 2024-05-06T00:28:05.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8799999999999998e-07 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812339+0000", 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2220000000000001e-06 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812345+0000", 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812350+0000", 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.724e-06 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817852+0000", 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005501751 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817860+0000", 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6580000000000002e-06 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11620 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802269+0000", 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34058543200000002, 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35598917000000002, 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11620 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.980 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802269+0000", 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802269+0000", 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802269+0000", 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3099999999999998e-07 2024-05-06T00:28:05.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802272+0000", 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.875e-06 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802279+0000", 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5599999999999999e-06 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802284+0000", 2024-05-06T00:28:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6239999999999999e-06 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803151+0000", 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086671800000000005 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803156+0000", 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.566e-06 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805699+0000", 2024-05-06T00:28:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025429979999999999 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805704+0000", 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5979999999999996e-06 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33199 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803233+0000", 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33962174899999997, 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35510468499999998, 2024-05-06T00:28:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33199 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803233+0000", 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803233+0000", 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803233+0000", 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7299999999999995e-07 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803236+0000", 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.604e-06 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803250+0000", 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3754000000000001e-05 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803255+0000", 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4210000000000003e-06 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803604+0000", 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034900799999999999 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803609+0000", 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.758e-06 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807196+0000", 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035872679999999998 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807203+0000", 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2640000000000002e-06 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.988 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33349 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805011+0000", 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33784344100000002, 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35340635999999997, 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33349 2024-05-06T00:28:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805011+0000", 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805011+0000", 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805011+0000", 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-07 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805027+0000", 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5591e-05 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805034+0000", 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1999999999999997e-06 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805039+0000", 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1449999999999999e-06 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808173+0000", 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031335809999999999 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808178+0000", 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2580000000000003e-06 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33426 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805938+0000", 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.336916836, 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35254718499999999, 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33426 2024-05-06T00:28:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805938+0000", 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805938+0000", 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805938+0000", 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1300000000000001e-07 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805940+0000", 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4739999999999999e-06 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805947+0000", 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0010000000000004e-06 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805953+0000", 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1229999999999999e-06 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809121+0000", 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003168101 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809127+0000", 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.959e-06 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33566 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810147+0000", 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33270699100000001, 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34840487199999998, 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33566 2024-05-06T00:28:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810147+0000", 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810147+0000", 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810148+0000", 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810150+0000", 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810155+0000", 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6019999999999999e-06 2024-05-06T00:28:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810160+0000", 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5990000000000004e-06 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812641+0000", 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024804670000000001 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8340000000000004e-06 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33618 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810820+0000", 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33203431700000002, 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34779955299999998, 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33618 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810820+0000", 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810820+0000", 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810820+0000", 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9099999999999999e-07 2024-05-06T00:28:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810828+0000", 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1339999999999998e-06 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810835+0000", 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5639999999999999e-06 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810840+0000", 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1410000000000004e-06 2024-05-06T00:28:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813635+0000", 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027950700000000002 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813641+0000", 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6620000000000002e-06 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33934 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811501+0000", 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33135347900000001, 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34718558500000002, 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33934 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811501+0000", 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811501+0000", 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811501+0000", 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811503+0000", 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0320000000000002e-06 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811510+0000", 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9469999999999997e-06 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811515+0000", 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7149999999999997e-06 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815258+0000", 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037433290000000001 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815265+0000", 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8650000000000003e-06 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34113 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.330348847, 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34624839499999999, 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34113 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2200000000000004e-07 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812509+0000", 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-06 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812519+0000", 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6979999999999993e-06 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812528+0000", 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5370000000000003e-06 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817889+0000", 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053609809999999999 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817903+0000", 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3545e-05 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11621 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802291+0000", 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34056375300000002, 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35657440000000001, 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11621 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802291+0000", 2024-05-06T00:28:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802291+0000", 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802291+0000", 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3099999999999998e-07 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802294+0000", 2024-05-06T00:28:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8569999999999999e-06 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802300+0000", 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2079999999999997e-06 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802305+0000", 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9189999999999998e-06 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803176+0000", 2024-05-06T00:28:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087125699999999996 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803181+0000", 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7899999999999999e-06 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805728+0000", 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025464730000000001 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805734+0000", 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4609999999999996e-06 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33201 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803263+0000", 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33959170599999999, 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35572715700000002, 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33201 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803263+0000", 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803263+0000", 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803263+0000", 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0900000000000002e-07 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803273+0000", 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0101e-05 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803281+0000", 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4889999999999998e-06 2024-05-06T00:28:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803290+0000", 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4350000000000003e-06 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803649+0000", 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035932299999999999 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803654+0000", 2024-05-06T00:28:06.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7389999999999999e-06 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807231+0000", 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003576549 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807236+0000", 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4389999999999999e-06 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33350 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805047+0000", 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337807684, 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.354024958, 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33350 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805047+0000", 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805047+0000", 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805047+0000", 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7e-07 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805049+0000", 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.322e-06 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805056+0000", 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3060000000000003e-06 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805064+0000", 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2709999999999992e-06 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808202+0000", 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031381400000000002 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808208+0000", 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3159999999999996e-06 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33428 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.027 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805959+0000", 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33689490500000002, 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35318022399999999, 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33428 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805959+0000", 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805959+0000", 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805960+0000", 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805962+0000", 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805969+0000", 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2280000000000003e-06 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805977+0000", 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4300000000000006e-06 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809152+0000", 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031746510000000001 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809157+0000", 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3519999999999997e-06 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33568 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810166+0000", 2024-05-06T00:28:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.332688598, 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34904551900000003, 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33568 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810166+0000", 2024-05-06T00:28:06.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810166+0000", 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810166+0000", 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810168+0000", 2024-05-06T00:28:06.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.81e-06 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810173+0000", 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.519e-06 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810180+0000", 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0400000000000004e-06 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812676+0000", 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024952360000000001 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812681+0000", 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7039999999999999e-06 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33620 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810847+0000", 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33200721100000002, 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.348439463, 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33620 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810847+0000", 2024-05-06T00:28:06.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810847+0000", 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810848+0000", 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5199999999999998e-07 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810850+0000", 2024-05-06T00:28:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1160000000000002e-06 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810855+0000", 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6919999999999999e-06 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810860+0000", 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7820000000000001e-06 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813666+0000", 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028063440000000001 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813671+0000", 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0250000000000002e-06 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33936 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.039 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811521+0000", 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33133321500000001, 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34783308499999999, 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33936 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811521+0000", 2024-05-06T00:28:06.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811521+0000", 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811522+0000", 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811524+0000", 2024-05-06T00:28:06.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9819999999999998e-06 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811531+0000", 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811536+0000", 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9919999999999998e-06 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815703+0000", 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041673819999999999 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815718+0000", 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4419e-05 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34114 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33031681099999999, 2024-05-06T00:28:06.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34688468099999997, 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34114 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-07 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812540+0000", 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4509999999999999e-06 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5069999999999997e-06 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812552+0000", 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817936+0000", 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053840650000000004 2024-05-06T00:28:06.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817944+0000", 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5890000000000003e-06 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11622 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802311+0000", 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34054301399999998, 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35757429000000002, 2024-05-06T00:28:06.047 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11622 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802311+0000", 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802311+0000", 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802312+0000", 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9099999999999999e-07 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802314+0000", 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6309999999999999e-06 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802321+0000", 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802326+0000", 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5249999999999996e-06 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803202+0000", 2024-05-06T00:28:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087625799999999998 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803207+0000", 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3900000000000003e-06 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805757+0000", 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025506119999999998 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805763+0000", 2024-05-06T00:28:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5040000000000002e-06 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33202 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.803337+0000", 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33951779199999998, 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35663796599999997, 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33202 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803337+0000", 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803337+0000", 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803337+0000", 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0500000000000004e-07 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803340+0000", 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0010000000000002e-06 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803348+0000", 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4789999999999999e-06 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803353+0000", 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8209999999999998e-06 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804076+0000", 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072213599999999996 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804081+0000", 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3600000000000004e-06 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807260+0000", 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031786679999999999 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807265+0000", 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1730000000000003e-06 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33352 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805070+0000", 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33778405099999997, 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35502393500000001, 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33352 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805070+0000", 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805070+0000", 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805071+0000", 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999998e-07 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805073+0000", 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.097e-06 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805079+0000", 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0469999999999999e-06 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805084+0000", 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.113e-06 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808232+0000", 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003147791 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808237+0000", 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2179999999999998e-06 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33430 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805986+0000", 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33686885900000002, 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35419473600000001, 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33430 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805986+0000", 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805986+0000", 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805986+0000", 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805988+0000", 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3499999999999999e-06 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805995+0000", 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.46e-06 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806000+0000", 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2870000000000004e-06 2024-05-06T00:28:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809182+0000", 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031816269999999998 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809187+0000", 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2240000000000003e-06 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33570 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810186+0000", 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.332668397, 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35006778100000002, 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33570 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810186+0000", 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810186+0000", 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810186+0000", 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3599999999999999e-07 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810188+0000", 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9740000000000001e-06 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810194+0000", 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5300000000000004e-06 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810198+0000", 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4900000000000002e-06 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812706+0000", 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025078790000000002 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812711+0000", 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1259999999999997e-06 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33622 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810867+0000", 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33198762700000001, 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.349457515, 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33622 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810867+0000", 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810867+0000", 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810867+0000", 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4799999999999999e-07 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810869+0000", 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.041e-06 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810875+0000", 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1260000000000003e-06 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810880+0000", 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813697+0000", 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00281679 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813704+0000", 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.737e-06 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33938 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811542+0000", 2024-05-06T00:28:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33131200999999999, 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.348849246, 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33938 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811542+0000", 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811542+0000", 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811543+0000", 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811545+0000", 2024-05-06T00:28:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2369999999999999e-06 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811553+0000", 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811557+0000", 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1019999999999999e-06 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815750+0000", 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041930070000000003 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.815758+0000", 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8800000000000008e-06 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34115 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33028170200000001, 2024-05-06T00:28:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34788950000000002, 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34115 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9700000000000002e-07 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812575+0000", 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.402e-06 2024-05-06T00:28:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812587+0000", 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7979999999999996e-06 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817971+0000", 2024-05-06T00:28:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053842719999999998 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817979+0000", 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4250000000000001e-06 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11623 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802333+0000", 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34052144400000001, 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.358197143, 2024-05-06T00:28:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11623 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802333+0000", 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802333+0000", 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802333+0000", 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0400000000000002e-07 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802336+0000", 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.959e-06 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802343+0000", 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3500000000000002e-06 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802348+0000", 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803228+0000", 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00088024200000000005 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803233+0000", 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.809e-06 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805786+0000", 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025528870000000002 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805791+0000", 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5330000000000002e-06 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33204 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804040+0000", 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33881487999999998, 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35657332899999999, 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33204 2024-05-06T00:28:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804040+0000", 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804040+0000", 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804040+0000", 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0200000000000001e-07 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804044+0000", 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999997e-06 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804055+0000", 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0716999999999999e-05 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804062+0000", 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1080000000000004e-06 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804104+0000", 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1752999999999998e-05 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804109+0000", 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6419999999999998e-06 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807290+0000", 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031813060000000001 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807296+0000", 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1040000000000003e-06 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33353 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805090+0000", 2024-05-06T00:28:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33776465500000002, 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35560563499999998, 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33353 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805090+0000", 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805090+0000", 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805090+0000", 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3000000000000002e-07 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805092+0000", 2024-05-06T00:28:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9250000000000002e-06 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805097+0000", 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4539999999999998e-06 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805103+0000", 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1220000000000003e-06 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808261+0000", 2024-05-06T00:28:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031587529999999998 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808267+0000", 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3850000000000001e-06 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33432 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.806008+0000", 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33684622600000003, 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35475467700000002, 2024-05-06T00:28:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33432 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806008+0000", 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806008+0000", 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806009+0000", 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806011+0000", 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806017+0000", 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3810000000000004e-06 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806022+0000", 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1680000000000003e-06 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809211+0000", 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031891570000000002 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809217+0000", 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4750000000000001e-06 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33572 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810204+0000", 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33265054700000002, 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35069024599999998, 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33572 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810204+0000", 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810204+0000", 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810204+0000", 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810206+0000", 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8670000000000001e-06 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810212+0000", 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5550000000000002e-06 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810216+0000", 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5650000000000003e-06 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812733+0000", 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025170140000000001 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812738+0000", 2024-05-06T00:28:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6439999999999999e-06 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33624 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810886+0000", 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33196837099999998, 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35008584500000001, 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33624 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810886+0000", 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810886+0000", 2024-05-06T00:28:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810886+0000", 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6199999999999999e-07 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810889+0000", 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2369999999999999e-06 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810896+0000", 2024-05-06T00:28:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9750000000000001e-06 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810902+0000", 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1659999999999999e-06 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813728+0000", 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028265550000000001 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813734+0000", 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6300000000000003e-06 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33940 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811563+0000", 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33129109000000001, 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34948222000000001, 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33940 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811563+0000", 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811563+0000", 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811564+0000", 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.77e-07 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811566+0000", 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0030000000000001e-06 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811571+0000", 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1309999999999997e-06 2024-05-06T00:28:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811575+0000", 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.014e-06 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816129+0000", 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545120000000001 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816137+0000", 2024-05-06T00:28:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.106e-06 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.330248712, 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.348511717, 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:28:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11624 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802354+0000", 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.340500213, 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35884136799999999, 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11624 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802354+0000", 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802354+0000", 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802355+0000", 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8299999999999998e-07 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802357+0000", 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.632e-06 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802363+0000", 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.246e-06 2024-05-06T00:28:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802369+0000", 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1370000000000001e-06 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803255+0000", 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000886223 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803260+0000", 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0010000000000001e-06 2024-05-06T00:28:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805814+0000", 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002553891 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805821+0000", 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0659999999999998e-06 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33206 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804074+0000", 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33878044499999999, 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35720540000000001, 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33206 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804074+0000", 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804074+0000", 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804075+0000", 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6599999999999996e-07 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804078+0000", 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9249999999999999e-06 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804085+0000", 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3649999999999998e-06 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804091+0000", 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6169999999999998e-06 2024-05-06T00:28:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804135+0000", 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4443999999999997e-05 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804139+0000", 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3290000000000004e-06 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807320+0000", 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003180259 2024-05-06T00:28:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807324+0000", 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33355 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805109+0000", 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337745146, 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35625612099999998, 2024-05-06T00:28:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33355 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805109+0000", 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805109+0000", 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805110+0000", 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9900000000000001e-07 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805112+0000", 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.187e-06 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805118+0000", 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2149999999999998e-06 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805123+0000", 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0900000000000004e-06 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808291+0000", 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031681460000000002 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808297+0000", 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2379999999999997e-06 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33434 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.806029+0000", 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33682579499999998, 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35540462, 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33434 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806029+0000", 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806029+0000", 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806029+0000", 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806044+0000", 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4656e-05 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806052+0000", 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5159999999999994e-06 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806057+0000", 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-06 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809241+0000", 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031846270000000002 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809247+0000", 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3129999999999998e-06 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33574 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810222+0000", 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33263279000000001, 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35128402399999997, 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33574 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810222+0000", 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810222+0000", 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810222+0000", 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2099999999999998e-07 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810224+0000", 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.874e-06 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810229+0000", 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4450000000000004e-06 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810234+0000", 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3919999999999996e-06 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812760+0000", 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002526525 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812765+0000", 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8189999999999999e-06 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33626 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810909+0000", 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33194505000000002, 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35066387100000002, 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33626 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810909+0000", 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810909+0000", 2024-05-06T00:28:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810910+0000", 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6100000000000002e-07 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810912+0000", 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.509e-06 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810919+0000", 2024-05-06T00:28:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9070000000000001e-06 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810924+0000", 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8709999999999996e-06 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813759+0000", 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002834661 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813764+0000", 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5949999999999998e-06 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33021734200000002, 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.34901632900000001, 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:28:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:28:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11625 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802375+0000", 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34047958299999997, 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35935236399999998, 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11625 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802375+0000", 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802375+0000", 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802375+0000", 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802378+0000", 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7159999999999999e-06 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802384+0000", 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-06T00:28:06.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802389+0000", 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2519999999999999e-06 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803282+0000", 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089314600000000004 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803287+0000", 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6619999999999996e-06 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805849+0000", 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025614800000000001 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805855+0000", 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.302e-06 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33208 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804098+0000", 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33875628400000002, 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35773515500000003, 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33208 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804098+0000", 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804098+0000", 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804099+0000", 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1100000000000001e-07 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804101+0000", 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804107+0000", 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999997e-06 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804112+0000", 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-06 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804161+0000", 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9190000000000002e-05 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804166+0000", 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2520000000000001e-06 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807348+0000", 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031823469999999999 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807353+0000", 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4399999999999996e-06 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33358 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805129+0000", 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33772544100000002, 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.356806085, 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33358 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805129+0000", 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805129+0000", 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805129+0000", 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805131+0000", 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805138+0000", 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0270000000000001e-06 2024-05-06T00:28:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805142+0000", 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4989999999999996e-06 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808320+0000", 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003178345 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808326+0000", 2024-05-06T00:28:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3260000000000003e-06 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33436 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.806979+0000", 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33587536600000001, 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35502465700000002, 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33436 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806979+0000", 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806979+0000", 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806980+0000", 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7700000000000004e-07 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.806984+0000", 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.048e-06 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807004+0000", 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0219000000000001e-05 2024-05-06T00:28:06.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807011+0000", 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3150000000000003e-06 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809271+0000", 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022594020000000002 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809277+0000", 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0589999999999999e-06 2024-05-06T00:28:06.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33576 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810239+0000", 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33261541100000003, 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35183671399999999, 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.159 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33576 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810239+0000", 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810239+0000", 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810239+0000", 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5699999999999999e-07 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810241+0000", 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.818e-06 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810247+0000", 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5369999999999997e-06 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810251+0000", 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5290000000000002e-06 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812787+0000", 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025360589999999998 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812792+0000", 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0849999999999996e-06 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.162 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33627 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810930+0000", 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33192395200000002, 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35121599999999997, 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33627 2024-05-06T00:28:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810930+0000", 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810930+0000", 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810931+0000", 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9200000000000002e-07 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810933+0000", 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.21e-06 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810940+0000", 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7800000000000003e-06 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810945+0000", 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8709999999999996e-06 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813790+0000", 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028454130000000002 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813795+0000", 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3890000000000004e-06 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33942 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811580+0000", 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.331273981, 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35063329300000001, 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33942 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811580+0000", 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811580+0000", 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811581+0000", 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0199999999999998e-07 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811586+0000", 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1410000000000004e-06 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811591+0000", 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1760000000000001e-06 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811595+0000", 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816162+0000", 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045674030000000003 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816169+0000", 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0820000000000001e-06 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33018355300000002, 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.349613544, 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11626 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802396+0000", 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34045870900000003, 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35995645300000001, 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11626 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802396+0000", 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802396+0000", 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802396+0000", 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.58e-07 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802399+0000", 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6469999999999998e-06 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802405+0000", 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2129999999999996e-06 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802410+0000", 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8409999999999999e-06 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803310+0000", 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089995899999999998 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803315+0000", 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8339999999999998e-06 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805877+0000", 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025625130000000002 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805883+0000", 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5489999999999997e-06 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33210 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804141+0000", 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33871352900000001, 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35834633300000002, 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33210 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804141+0000", 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804141+0000", 2024-05-06T00:28:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804141+0000", 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.89e-07 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804144+0000", 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6699999999999998e-06 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804152+0000", 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5569999999999998e-06 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804157+0000", 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5729999999999998e-06 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804187+0000", 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9366999999999999e-05 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804191+0000", 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4050000000000002e-06 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807379+0000", 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031877369999999999 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807385+0000", 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9490000000000001e-06 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33361 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805149+0000", 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33770586600000002, 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35743783200000001, 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33361 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805149+0000", 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805149+0000", 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805149+0000", 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8099999999999998e-07 2024-05-06T00:28:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805151+0000", 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2689999999999998e-06 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805157+0000", 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0240000000000003e-06 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805162+0000", 2024-05-06T00:28:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9269999999999996e-06 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808350+0000", 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031879790000000001 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808356+0000", 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5040000000000002e-06 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33438 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.807024+0000", 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33583069999999998, 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35563172300000001, 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33438 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807024+0000", 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807024+0000", 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807024+0000", 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-05-06T00:28:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807027+0000", 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.022e-06 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807035+0000", 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5689999999999999e-06 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807050+0000", 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5330999999999999e-05 2024-05-06T00:28:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809301+0000", 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022511430000000002 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809307+0000", 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2190000000000003e-06 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33577 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810257+0000", 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33259766000000002, 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35248088, 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33577 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810257+0000", 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810257+0000", 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810257+0000", 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0100000000000001e-07 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810259+0000", 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1880000000000001e-06 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810265+0000", 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810270+0000", 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9099999999999996e-06 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812818+0000", 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025482370000000001 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812823+0000", 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33628 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810952+0000", 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33190285899999999, 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35186455599999999, 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33628 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810952+0000", 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810952+0000", 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810952+0000", 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3599999999999999e-07 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810954+0000", 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810959+0000", 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4600000000000002e-06 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810964+0000", 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7990000000000001e-06 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813822+0000", 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028579550000000001 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813828+0000", 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3360000000000002e-06 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33944 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811600+0000", 2024-05-06T00:28:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33125392100000001, 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35129247000000002, 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33944 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811600+0000", 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811600+0000", 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811601+0000", 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811603+0000", 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7430000000000001e-06 2024-05-06T00:28:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811608+0000", 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1250000000000001e-06 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811613+0000", 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7230000000000001e-06 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816191+0000", 2024-05-06T00:28:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004577897 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816198+0000", 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.832e-06 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33000309700000002, 2024-05-06T00:28:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35011239900000002, 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:11627 3.10 3.f419e410 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.802416+0000", 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.34043850199999998, 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.36061580399999998, 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11627 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802416+0000", 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802416+0000", 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802416+0000", 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3799999999999998e-07 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802419+0000", 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7049999999999999e-06 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802425+0000", 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2469999999999997e-06 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.802430+0000", 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1379999999999998e-06 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803347+0000", 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091640700000000005 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.803352+0000", 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0350000000000001e-06 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805902+0000", 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025506489999999999 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805909+0000", 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9249999999999998e-06 2024-05-06T00:28:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33212 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.804165+0000", 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33868903500000003, 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35894896399999998, 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33212 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804165+0000", 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804165+0000", 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804166+0000", 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3500000000000002e-07 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804168+0000", 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1449999999999998e-06 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804175+0000", 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6150000000000002e-06 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804180+0000", 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804212+0000", 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1992000000000001e-05 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.804216+0000", 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4209999999999997e-06 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807410+0000", 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031938209999999999 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807416+0000", 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7169999999999996e-06 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33362 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.805169+0000", 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.337685656, 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35802836799999999, 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33362 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805169+0000", 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805169+0000", 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805169+0000", 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805171+0000", 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1210000000000001e-06 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805177+0000", 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0360000000000003e-06 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.805182+0000", 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808381+0000", 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031985160000000002 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.808386+0000", 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5990000000000001e-06 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33440 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.807059+0000", 2024-05-06T00:28:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33579537599999998, 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.356205989, 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33440 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807059+0000", 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807059+0000", 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807059+0000", 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807062+0000", 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.931e-06 2024-05-06T00:28:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807069+0000", 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7120000000000003e-06 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.807074+0000", 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1240000000000004e-06 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809331+0000", 2024-05-06T00:28:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022567659999999999 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.809336+0000", 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2229999999999998e-06 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33578 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810276+0000", 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33257798799999999, 2024-05-06T00:28:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35306174000000001, 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33578 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810276+0000", 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810276+0000", 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810277+0000", 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810279+0000", 2024-05-06T00:28:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0990000000000001e-06 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810285+0000", 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0349999999999998e-06 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810290+0000", 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7759999999999997e-06 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812847+0000", 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025574719999999999 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9049999999999996e-06 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33630 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.810971+0000", 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.33188379200000001, 2024-05-06T00:28:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35244542600000001, 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33630 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810971+0000", 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810971+0000", 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810971+0000", 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.53e-07 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810973+0000", 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.063e-06 2024-05-06T00:28:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810980+0000", 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960000000000002e-06 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.810985+0000", 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813851+0000", 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002865537 2024-05-06T00:28:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813856+0000", 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3890000000000004e-06 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:33945 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.811620+0000", 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.331234578, 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35187916299999999, 2024-05-06T00:28:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 33945 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811620+0000", 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811620+0000", 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811620+0000", 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2000000000000001e-07 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811622+0000", 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2479999999999999e-06 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811628+0000", 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4689999999999996e-06 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.811633+0000", 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6909999999999996e-06 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816222+0000", 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045893150000000001 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.816229+0000", 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0319999999999998e-06 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3.3dc6b552 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.329969454, 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.35070807999999998, 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 256 2024-05-06T00:28:06.243 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:06.244 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:06.244 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:05.389+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:06.244 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:06.245 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:06.245+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:06.348 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:06.345+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:06.384 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:06.384 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:06.401 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:06.410 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:06.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:06.427 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:06.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:06.443 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:06.448 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:06.445+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:06.464 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:06.465 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:06.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:06.486 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:06.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:06.507 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:06.525 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:06.525 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:06.539 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:06.548 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:06.547+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:06.649 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:06.647+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:06.698 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:06.698 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:06.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:06.750 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:06.751+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:06.790 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:06.790 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:06.790 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.9229375911363212 2024-05-06T00:28:06.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 0.9229375911363212 2024-05-06T00:28:06.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:28:06.862 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:06.862 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:06.862 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:06.861+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:06.879 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:06.964 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:06.965+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:07.053 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:07.051+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:07.061 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:07.061 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:07.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:07.154 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:07.151+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:07.233 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:07.233 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:07.247 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:07.256 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:07.251+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:07.272 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-05-06T00:28:07.357 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:07.357+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:07.360 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:07.360 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:07.372 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:07.464 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:07.465+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:07.529 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:07.529 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:07.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:07.585 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:07.585+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:07.659 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:07.655+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:07.675 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:07.675 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:07.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:07.755 INFO:teuthology.orchestra.run.smithi186.stdout:2987 write 0xe7ea800 thru 0xe7eac7d (0x47e bytes) 2024-05-06T00:28:07.755 INFO:teuthology.orchestra.run.smithi186.stdout:2988 write 0x4fa0800 thru 0x4fafdee (0xf5ef bytes) 2024-05-06T00:28:07.755 INFO:teuthology.orchestra.run.smithi186.stdout:2992 write 0xd1f2200 thru 0xd1fb763 (0x9564 bytes) 2024-05-06T00:28:07.755 INFO:teuthology.orchestra.run.smithi186.stdout:2995 punch from 0x1164c00 to 0x1168a00, (0x3e00 bytes) 2024-05-06T00:28:07.777 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:07.775+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:07.802 INFO:teuthology.orchestra.run.smithi186.stdout:2997 read 0x7e60200 thru 0x7e69840 (0x9641 bytes) 2024-05-06T00:28:07.807 INFO:teuthology.orchestra.run.smithi186.stdout:2998 read 0x5cb9000 thru 0x5cbcf65 (0x3f66 bytes) 2024-05-06T00:28:07.850 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:28:07.860 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:07.857+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:07.871 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:07.871 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:07.879 INFO:teuthology.orchestra.run.smithi186.stdout:3000 trunc from 0xe7eae00 to 0x19c3e00 2024-05-06T00:28:07.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:07.962 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:07.961+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:08.063 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:08.059+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:08.063 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:08.064 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:08.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:08.163 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:08.161+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:08.238 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:08.238 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:08.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:08.264 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:08.265+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:08.366 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:08.363+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:08.426 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:08.426 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:08.443 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:08.466 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:08.465+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:08.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:08.570 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:08.569+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:08.640 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:08.640 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:08.658 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:08.705 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:08.703+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:08.746 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:08.773 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:08.769+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:08.787 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:08.787 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:08.791 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:08.792 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:08.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:08.842 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.1 to 0.922938 (ec45) 2024-05-06T00:28:08.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:08.854 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:08.869 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:08.869+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:08.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:08.879 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:08.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:08.915 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:08.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:08.940 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:08.966 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:08.966 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:08.966 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:08.966 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:08.966 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:08.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:09.001 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:09.001+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:09.071 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:09.069+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:09.112 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:09.112 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:09.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:09.112 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:09.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.112 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.651393351000003, 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:28:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:28:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:28:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.650591362, 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:28:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.650504908000002, 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:28:09.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.650432315000003, 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:28:09.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:28:09.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:09.131 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.650372464, 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:28:09.135 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.650296781000002, 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.136 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:28:09.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:28:09.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2967 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 159744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.642672793000003, 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026274390000000002, 2024-05-06T00:28:09.140 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2967 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467898+0000", 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.046e-06 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467917+0000", 2024-05-06T00:28:09.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467937+0000", 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9613999999999999e-05 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467961+0000", 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.419e-05 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.493935+0000", 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025973922 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494053+0000", 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011866099999999999 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494171+0000", 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011806099999999999 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.642591207999999, 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:28:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:28:09.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.642537001000001, 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:28:09.149 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:28:09.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:28:09.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.642484852000003, 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:28:09.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:28:09.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3040 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.640629529999998, 2024-05-06T00:28:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026298018999999999, 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3040 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469941+0000", 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0979999999999999e-06 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469946+0000", 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0170000000000004e-06 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469963+0000", 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6541999999999998e-05 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469977+0000", 2024-05-06T00:28:09.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4015999999999999e-05 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495954+0000", 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025976902999999999 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496101+0000", 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147529 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496238+0000", 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136914 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.640527136999999, 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:28:09.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.640450797, 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:28:09.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.825313090000002, 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.172 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:28:09.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:28:09.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:09.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.825260649000001, 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:28:09.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.818240666000001, 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:09.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:28:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:28:09.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.818193954999998, 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.190 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:28:09.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.817863846999998, 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.817835924999997, 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:28:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.806969035999998, 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:09.210 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:09.212 INFO:teuthology.orchestra.run.smithi186.stdout:3007 write 0xf20c00 thru 0xf2f5fd (0xe9fe bytes) 2024-05-06T00:28:09.212 INFO:teuthology.orchestra.run.smithi186.stdout:3009 read 0x15ea000 thru 0x15ee73f (0x4740 bytes) 2024-05-06T00:28:09.212 INFO:teuthology.orchestra.run.smithi186.stdout:3010 read 0x970e00 thru 0x97ff91 (0xf192 bytes) 2024-05-06T00:28:09.213 INFO:teuthology.orchestra.run.smithi186.stdout:3012 write 0x68f400 thru 0x691b27 (0x2728 bytes) 2024-05-06T00:28:09.213 INFO:teuthology.orchestra.run.smithi186.stdout:3015 write 0x77c7000 thru 0x77cd549 (0x654a bytes) 2024-05-06T00:28:09.213 INFO:teuthology.orchestra.run.smithi186.stdout:3016 trunc from 0x77cd600 to 0x2cde600 2024-05-06T00:28:09.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:09.214 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:09.209+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:09.263 INFO:teuthology.orchestra.run.smithi186.stdout:3017 write 0xccc1e00 thru 0xccc201c (0x21d bytes) 2024-05-06T00:28:09.265 INFO:teuthology.orchestra.run.smithi186.stdout:3020 write 0x204ea00 thru 0x205672f (0x7d30 bytes) 2024-05-06T00:28:09.267 INFO:teuthology.orchestra.run.smithi186.stdout:3021 read 0x2fc3a00 thru 0x2fc8fe9 (0x55ea bytes) 2024-05-06T00:28:09.268 INFO:teuthology.orchestra.run.smithi186.stdout:3026 write 0x9cc6e00 thru 0x9cd61d3 (0xf3d4 bytes) 2024-05-06T00:28:09.304 INFO:teuthology.orchestra.run.smithi186.stdout:3028 trunc from 0xccc2200 to 0xd92000 2024-05-06T00:28:09.314 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:09.311+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:09.355 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:09.355 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:09.355 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:09.357 INFO:teuthology.orchestra.run.smithi186.stdout:3031 read 0x327200 thru 0x3303a8 (0x91a9 bytes) 2024-05-06T00:28:09.362 INFO:teuthology.orchestra.run.smithi186.stdout:3032 write 0xbb1000 thru 0xbbc418 (0xb419 bytes) 2024-05-06T00:28:09.362 INFO:teuthology.orchestra.run.smithi186.stdout:3034 read 0xc11200 thru 0xc17f6f (0x6d70 bytes) 2024-05-06T00:28:09.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:09.395 INFO:teuthology.orchestra.run.smithi186.stdout:3035 trunc from 0xd92000 to 0xd427800 2024-05-06T00:28:09.412 INFO:teuthology.orchestra.run.smithi186.stdout:3037 write 0xcf41000 thru 0xcf4d7d9 (0xc7da bytes) 2024-05-06T00:28:09.415 INFO:teuthology.orchestra.run.smithi186.stdout:3038 write 0xcded000 thru 0xcdf41f6 (0x71f7 bytes) 2024-05-06T00:28:09.415 INFO:teuthology.orchestra.run.smithi186.stdout:3039 read 0x69c5800 thru 0x69d4eda (0xf6db bytes) 2024-05-06T00:28:09.415 INFO:teuthology.orchestra.run.smithi186.stdout:3043 read 0x9fab600 thru 0x9fb7349 (0xbd4a bytes) 2024-05-06T00:28:09.415 INFO:teuthology.orchestra.run.smithi186.stdout:3045 punch from 0x4c48e00 to 0x4c57200, (0xe400 bytes) 2024-05-06T00:28:09.417 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:09.417+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:09.417 INFO:teuthology.orchestra.run.smithi186.stdout:3047 write 0xa3df800 thru 0xa3e327d (0x3a7e bytes) 2024-05-06T00:28:09.417 INFO:teuthology.orchestra.run.smithi186.stdout:3048 write 0x93a5a00 thru 0x93b11ec (0xb7ed bytes) 2024-05-06T00:28:09.417 INFO:teuthology.orchestra.run.smithi186.stdout:3049 read 0x9ca5200 thru 0x9ca918e (0x3f8f bytes) 2024-05-06T00:28:09.417 INFO:teuthology.orchestra.run.smithi186.stdout:3050 write 0xa539a00 thru 0xa53dd14 (0x4315 bytes) 2024-05-06T00:28:09.433 INFO:teuthology.orchestra.run.smithi186.stdout:3051 trunc from 0xd427800 to 0x40aca00 2024-05-06T00:28:09.472 INFO:teuthology.orchestra.run.smithi186.stdout:3052 write 0x300ec00 thru 0x300f315 (0x716 bytes) 2024-05-06T00:28:09.486 INFO:teuthology.orchestra.run.smithi186.stdout:3054 trunc from 0x40aca00 to 0x9515e00 2024-05-06T00:28:09.500 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:09.500 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:09.515 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:09.515 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:09.513+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:09.531 INFO:teuthology.orchestra.run.smithi186.stdout:3055 trunc from 0x9515e00 to 0x260000 2024-05-06T00:28:09.568 INFO:teuthology.orchestra.run.smithi186.stdout:3065 read 0x43a00 thru 0x4a19d (0x679e bytes) 2024-05-06T00:28:09.571 INFO:teuthology.orchestra.run.smithi186.stdout:3068 read 0x250c00 thru 0x251962 (0xd63 bytes) 2024-05-06T00:28:09.616 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:09.613+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.880323645000004, 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:28:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:28:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:28:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:28:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:28:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:28:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.880307442000003, 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:28:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34113 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 845824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8721773600000002, 2024-05-06T00:28:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.073299384, 2024-05-06T00:28:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34113 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2200000000000004e-07 2024-05-06T00:28:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812509+0000", 2024-05-06T00:28:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-06 2024-05-06T00:28:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812519+0000", 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6979999999999993e-06 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812528+0000", 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5370000000000003e-06 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817889+0000", 2024-05-06T00:28:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053609809999999999 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817903+0000", 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3545e-05 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885659+0000", 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.067755797 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885734+0000", 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5698999999999995e-05 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885805+0000", 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0442000000000001e-05 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34114 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 851968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8721453239999999, 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0734813590000001, 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34114 2024-05-06T00:28:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-07 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812540+0000", 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4509999999999999e-06 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5069999999999997e-06 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812552+0000", 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817936+0000", 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053840650000000004 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817944+0000", 2024-05-06T00:28:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5890000000000003e-06 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885879+0000", 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0679342119999999 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885952+0000", 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2924999999999997e-05 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886019+0000", 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7358000000000001e-05 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34115 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 858112~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8721102150000002, 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0736428419999999, 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34115 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9700000000000002e-07 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812575+0000", 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.402e-06 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812587+0000", 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7979999999999996e-06 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817971+0000", 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053842719999999998 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817979+0000", 2024-05-06T00:28:09.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4250000000000001e-06 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886063+0000", 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068084475 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886129+0000", 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6109e-05 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886216+0000", 2024-05-06T00:28:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6312999999999998e-05 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.872077225, 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:09.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:28:09.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8720458549999996, 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.728 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:09.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8720120659999999, 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.733 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:28:09.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8718316100000001, 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:28:09.738 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:28:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.871797967, 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:28:09.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:28:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8717741849999996, 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:28:09.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:28:09.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:28:09.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8717479079999997, 2024-05-06T00:28:09.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:28:09.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:28:09.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8717046670000004, 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.760 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:28:09.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:28:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:28:09.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:28:09.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8716750380000002, 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:09.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.871642188, 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:28:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.774 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.774 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.774 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.774 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.774 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.774 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:28:09.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.775 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:09.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:09.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:28:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:28:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8716105499999998, 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:28:09.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.871577748, 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.792 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8715457149999999, 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.797 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8715136359999995, 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:09.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:09.803 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8714823279999999, 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:28:09.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:09.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:28:09.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:28:09.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.808 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:09.808 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:09.808 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:09.809 INFO:teuthology.orchestra.run.smithi186.stdout:3073 write 0x4a89a00 thru 0x4a90209 (0x680a bytes) 2024-05-06T00:28:09.809 INFO:teuthology.orchestra.run.smithi186.stdout:3074 punch from 0x1f43400 to 0x1f45c00, (0x2800 bytes) 2024-05-06T00:28:09.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:09.810 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:09.809+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:09.832 INFO:teuthology.orchestra.run.smithi186.stdout:3075 write 0x8413a00 thru 0x8419fb2 (0x65b3 bytes) 2024-05-06T00:28:09.834 INFO:teuthology.orchestra.run.smithi186.stdout:3078 punch from 0x7cbac00 to 0x7cc3200, (0x8600 bytes) 2024-05-06T00:28:09.835 INFO:teuthology.orchestra.run.smithi186.stdout:3079 read 0x2d2e600 thru 0x2d3a935 (0xc336 bytes) 2024-05-06T00:28:09.836 INFO:teuthology.orchestra.run.smithi186.stdout:3081 punch from 0x4fef400 to 0x4fefe00, (0xa00 bytes) 2024-05-06T00:28:09.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:28:09.876 INFO:teuthology.orchestra.run.smithi186.stdout:3082 write 0xe1c5e00 thru 0xe1cf6d3 (0x98d4 bytes) 2024-05-06T00:28:09.878 INFO:teuthology.orchestra.run.smithi186.stdout:3084 read 0x5d51600 thru 0x5d56970 (0x5371 bytes) 2024-05-06T00:28:09.878 INFO:teuthology.orchestra.run.smithi186.stdout:3085 read 0x2bbbe00 thru 0x2bc0d8b (0x4f8c bytes) 2024-05-06T00:28:09.879 INFO:teuthology.orchestra.run.smithi186.stdout:3087 punch from 0xd950e00 to 0xd952200, (0x1400 bytes) 2024-05-06T00:28:09.880 INFO:teuthology.orchestra.run.smithi186.stdout:3088 read 0x4e36000 thru 0x4e3afa6 (0x4fa7 bytes) 2024-05-06T00:28:09.910 INFO:teuthology.orchestra.run.smithi186.stdout:3090 write 0xe776e00 thru 0xe77b992 (0x4b93 bytes) 2024-05-06T00:28:09.912 INFO:teuthology.orchestra.run.smithi186.stdout:3092 read 0x498a400 thru 0x498ddb9 (0x39ba bytes) 2024-05-06T00:28:09.912 INFO:teuthology.orchestra.run.smithi186.stdout:3093 read 0x51cec00 thru 0x51d55b0 (0x69b1 bytes) 2024-05-06T00:28:09.913 INFO:teuthology.orchestra.run.smithi186.stdout:3098 write 0x4466a00 thru 0x4472fca (0xc5cb bytes) 2024-05-06T00:28:09.913 INFO:teuthology.orchestra.run.smithi186.stdout:3100 write 0xad20000 thru 0xad2f3d3 (0xf3d4 bytes) 2024-05-06T00:28:09.913 INFO:teuthology.orchestra.run.smithi186.stdout:3101 read 0x1ad1a00 thru 0x1adabb6 (0x91b7 bytes) 2024-05-06T00:28:09.914 INFO:teuthology.orchestra.run.smithi186.stdout:3102 punch from 0x3454000 to 0x3458600, (0x4600 bytes) 2024-05-06T00:28:09.915 INFO:teuthology.orchestra.run.smithi186.stdout:3103 read 0x5011600 thru 0x5014f01 (0x3902 bytes) 2024-05-06T00:28:09.915 INFO:teuthology.orchestra.run.smithi186.stdout:3105 write 0xdbcf200 thru 0xdbd311f (0x3f20 bytes) 2024-05-06T00:28:09.916 INFO:teuthology.orchestra.run.smithi186.stdout:3108 write 0x2d4c600 thru 0x2d51f84 (0x5985 bytes) 2024-05-06T00:28:09.918 INFO:teuthology.orchestra.run.smithi186.stdout:3110 write 0x3988800 thru 0x3996eec (0xe6ed bytes) 2024-05-06T00:28:09.918 INFO:teuthology.orchestra.run.smithi186.stdout:3114 read 0xab99600 thru 0xab9c158 (0x2b59 bytes) 2024-05-06T00:28:09.934 INFO:teuthology.orchestra.run.smithi186.stdout:3115 trunc from 0xe77ba00 to 0x4f3e400 2024-05-06T00:28:09.940 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:09.941+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:09.973 INFO:teuthology.orchestra.run.smithi186.stdout:3116 trunc from 0x4f3e400 to 0x1629400 2024-05-06T00:28:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:09.998 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:09.998 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:09.998 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:10.011 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:10.007+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:10.014 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:10.022 INFO:teuthology.orchestra.run.smithi186.stdout:3118 trunc from 0x1629400 to 0xb8c9800 2024-05-06T00:28:10.035 INFO:teuthology.orchestra.run.smithi186.stdout:3119 trunc from 0xb8c9800 to 0xb1ef000 2024-05-06T00:28:10.051 INFO:teuthology.orchestra.run.smithi186.stdout:3120 trunc from 0xb1ef000 to 0xac00a00 2024-05-06T00:28:10.065 INFO:teuthology.orchestra.run.smithi186.stdout:3122 read 0x250800 thru 0x25dd1f (0xd520 bytes) 2024-05-06T00:28:10.068 INFO:teuthology.orchestra.run.smithi186.stdout:3123 trunc from 0xac00a00 to 0xa9f3800 2024-05-06T00:28:10.083 INFO:teuthology.orchestra.run.smithi186.stdout:3124 punch from 0x95a000 to 0x967e00, (0xde00 bytes) 2024-05-06T00:28:10.085 INFO:teuthology.orchestra.run.smithi186.stdout:3125 read 0x850de00 thru 0x851c896 (0xea97 bytes) 2024-05-06T00:28:10.089 INFO:teuthology.orchestra.run.smithi186.stdout:3127 trunc from 0xa9f3800 to 0x9627e00 2024-05-06T00:28:10.109 INFO:teuthology.orchestra.run.smithi186.stdout:3130 trunc from 0x9627e00 to 0xc452c00 2024-05-06T00:28:10.112 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:10.107+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:10.124 INFO:teuthology.orchestra.run.smithi186.stdout:3131 read 0x399f400 thru 0x39aba1b (0xc61c bytes) 2024-05-06T00:28:10.146 INFO:teuthology.orchestra.run.smithi186.stdout:3132 trunc from 0xc452c00 to 0x3479000 2024-05-06T00:28:10.182 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:10.183 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:10.183 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:10.183 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:10.183 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:10.189 INFO:teuthology.orchestra.run.smithi186.stdout:3133 write 0x8ab9400 thru 0x8ab9c18 (0x819 bytes) 2024-05-06T00:28:10.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:10.213 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:10.211+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:10.224 INFO:teuthology.orchestra.run.smithi186.stdout:3134 write 0xdcb2400 thru 0xdcc067d (0xe27e bytes) 2024-05-06T00:28:10.226 INFO:teuthology.orchestra.run.smithi186.stdout:3136 read 0xc94fc00 thru 0xc94fe55 (0x256 bytes) 2024-05-06T00:28:10.226 INFO:teuthology.orchestra.run.smithi186.stdout:3137 write 0x703c200 thru 0x7045ced (0x9aee bytes) 2024-05-06T00:28:10.226 INFO:teuthology.orchestra.run.smithi186.stdout:3139 write 0xd3fb600 thru 0xd4047f7 (0x91f8 bytes) 2024-05-06T00:28:10.226 INFO:teuthology.orchestra.run.smithi186.stdout:3142 read 0x1379000 thru 0x1383836 (0xa837 bytes) 2024-05-06T00:28:10.228 INFO:teuthology.orchestra.run.smithi186.stdout:3145 punch from 0xaec1c00 to 0xaed0600, (0xea00 bytes) 2024-05-06T00:28:10.229 INFO:teuthology.orchestra.run.smithi186.stdout:3147 punch from 0x492d400 to 0x4934200, (0x6e00 bytes) 2024-05-06T00:28:10.235 INFO:teuthology.orchestra.run.smithi186.stdout:3148 trunc from 0xdcc0800 to 0xec99c00 2024-05-06T00:28:10.258 INFO:teuthology.orchestra.run.smithi186.stdout:3150 punch from 0x177600 to 0x181c00, (0xa600 bytes) 2024-05-06T00:28:10.281 INFO:teuthology.orchestra.run.smithi186.stdout:3151 trunc from 0xec99c00 to 0x6824000 2024-05-06T00:28:10.307 INFO:teuthology.orchestra.run.smithi186.stdout:3152 punch from 0x1e1f000 to 0x1e2a800, (0xb800 bytes) 2024-05-06T00:28:10.313 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:10.313+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:10.328 INFO:teuthology.orchestra.run.smithi186.stdout:3156 write 0x8e26600 thru 0x8e27ee6 (0x18e7 bytes) 2024-05-06T00:28:10.336 INFO:teuthology.orchestra.run.smithi186.stdout:3159 trunc from 0x8e28000 to 0xbf6da00 2024-05-06T00:28:10.338 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21566 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 417792~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3104923929999996, 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090303018999999998, 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.339 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21566 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5400000000000002e-07 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026960+0000", 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4449999999999999e-06 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026966+0000", 2024-05-06T00:28:10.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6200000000000001e-06 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026978+0000", 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1248e-05 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116885+0000", 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.089907163999999998 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116986+0000", 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101363 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117255+0000", 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026882499999999999 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21568 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 423936~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.343 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3103849079999996, 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090403081999999996, 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21568 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027060+0000", 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027063+0000", 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3249999999999999e-06 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027072+0000", 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2859999999999999e-06 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027078+0000", 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117286+0000", 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090207976999999995 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117372+0000", 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5865000000000005e-05 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117463+0000", 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0662999999999996e-05 2024-05-06T00:28:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21570 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 430080~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3103582989999998, 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090545366000000002, 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21570 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027087+0000", 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3799999999999998e-07 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027089+0000", 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.12e-06 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027095+0000", 2024-05-06T00:28:10.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4590000000000003e-06 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027100+0000", 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117492+0000", 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090391632999999999 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117546+0000", 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4177e-05 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117631+0000", 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5173999999999995e-05 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21572 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 436224~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.352 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3103359379999997, 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090693918999999998, 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21572 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027109+0000", 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7100000000000002e-07 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027112+0000", 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5490000000000001e-06 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027119+0000", 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7829999999999999e-06 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027124+0000", 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.04e-06 2024-05-06T00:28:10.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117662+0000", 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090537113000000002 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117717+0000", 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5798999999999999e-05 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117802+0000", 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5063999999999997e-05 2024-05-06T00:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3103118719999998, 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:10.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3102910830000001, 2024-05-06T00:28:10.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3102710210000001, 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:28:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3102517450000004, 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:28:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3102316270000003, 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.379 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3102002180000003, 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:28:10.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3101673060000003, 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.384 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:28:10.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3101467190000005, 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.389 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3101273390000001, 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:28:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:28:10.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.398 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3100826579999998, 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:28:10.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:28:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.310017567, 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:10.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.407 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3099859929999997, 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:10.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.412 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3099637319999999, 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:28:10.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.3099426159999998, 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:28:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6522259899999998, 2024-05-06T00:28:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:28:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6518157479999998, 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:28:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.430 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:10.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:10.436 INFO:teuthology.orchestra.run.smithi186.stdout:3162 write 0x17cc00 thru 0x188192 (0xb593 bytes) 2024-05-06T00:28:10.436 INFO:teuthology.orchestra.run.smithi186.stdout:3163 write 0x4745400 thru 0x474644f (0x1050 bytes) 2024-05-06T00:28:10.436 INFO:teuthology.orchestra.run.smithi186.stdout:3164 punch from 0x24fb200 to 0x2504c00, (0x9a00 bytes) 2024-05-06T00:28:10.436 INFO:teuthology.orchestra.run.smithi186.stdout:3165 write 0x3c92200 thru 0x3ca0776 (0xe577 bytes) 2024-05-06T00:28:10.436 INFO:teuthology.orchestra.run.smithi186.stdout:3166 punch from 0x15eaa00 to 0x15eea00, (0x4000 bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3167 read 0x96e1c00 thru 0x96f07eb (0xebec bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3169 write 0x3769c00 thru 0x377094e (0x6d4f bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3170 read 0x4f8c400 thru 0x4f99238 (0xce39 bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3171 read 0x9669800 thru 0x966de37 (0x4638 bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3172 write 0x4fe5000 thru 0x4ff1266 (0xc267 bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3173 write 0xd24a00 thru 0xd29c5f (0x5260 bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3175 write 0xdfaac00 thru 0xdfabe63 (0x1264 bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3176 read 0x2e8c00 thru 0x2e9ac2 (0xec3 bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3177 read 0x5adf800 thru 0x5aea8ba (0xb0bb bytes) 2024-05-06T00:28:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:3178 read 0x6da5400 thru 0x6da8659 (0x325a bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3179 write 0xe3bc00 thru 0xe48f88 (0xd389 bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3180 read 0xd1f0c00 thru 0xd1fe59f (0xd9a0 bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3186 punch from 0x33c2c00 to 0x33c8600, (0x5a00 bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3187 write 0xd364000 thru 0xd3676ea (0x36eb bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3188 write 0x4024200 thru 0x403089e (0xc69f bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3190 punch from 0xafd9e00 to 0xafe4200, (0xa400 bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3191 write 0x76cc200 thru 0x76dab73 (0xe974 bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3192 write 0xa936200 thru 0xa93e0e4 (0x7ee5 bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3197 read 0x8f8ca00 thru 0x8f8ce9b (0x49c bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3199 write 0x48f7000 thru 0x4903dad (0xcdae bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3201 write 0xed3a00 thru 0xedf012 (0xb613 bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3202 write 0x19bfc00 thru 0x19c956e (0x996f bytes) 2024-05-06T00:28:10.438 INFO:teuthology.orchestra.run.smithi186.stdout:3205 write 0x1e94600 thru 0x1e9c0d1 (0x7ad2 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3206 read 0xc964a00 thru 0xc969143 (0x4744 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3208 write 0x71b3400 thru 0x71bd397 (0x9f98 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3210 read 0xcde00 thru 0xd4469 (0x666a bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3212 read 0x8b36600 thru 0x8b3d284 (0x6c85 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3213 write 0x95f6600 thru 0x95fffe2 (0x99e3 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3214 punch from 0xd463200 to 0xd472200, (0xf000 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3220 read 0xc99f400 thru 0xc9ade08 (0xea09 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3221 read 0x337a200 thru 0x337fa2d (0x582e bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3222 punch from 0xa16600 to 0xa1b600, (0x5000 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3223 punch from 0x3e5da00 to 0x3e6c000, (0xe600 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3224 punch from 0x14ee200 to 0x14f3200, (0x5000 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3230 write 0x7749a00 thru 0x775166f (0x7c70 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3231 write 0x866b000 thru 0x8676cff (0xbd00 bytes) 2024-05-06T00:28:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:3233 punch from 0x677b200 to 0x6788800, (0xd600 bytes) 2024-05-06T00:28:10.440 INFO:teuthology.orchestra.run.smithi186.stdout:3235 punch from 0x4afa200 to 0x4afa800, (0x600 bytes) 2024-05-06T00:28:10.440 INFO:teuthology.orchestra.run.smithi186.stdout:3236 trunc from 0xdfac000 to 0x4465e00 2024-05-06T00:28:10.440 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:10.427+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:10.480 INFO:teuthology.orchestra.run.smithi186.stdout:3237 write 0xbc06000 thru 0xbc14f28 (0xef29 bytes) 2024-05-06T00:28:10.481 INFO:teuthology.orchestra.run.smithi186.stdout:3239 write 0xbac4600 thru 0xbacc19c (0x7b9d bytes) 2024-05-06T00:28:10.482 INFO:teuthology.orchestra.run.smithi186.stdout:3241 trunc from 0xbc15000 to 0xbe39600 2024-05-06T00:28:10.504 INFO:teuthology.orchestra.run.smithi186.stdout:3242 punch from 0x14b5800 to 0x14c4000, (0xe800 bytes) 2024-05-06T00:28:10.506 INFO:teuthology.orchestra.run.smithi186.stdout:3243 read 0x18c5000 thru 0x18cbfa9 (0x6faa bytes) 2024-05-06T00:28:10.507 INFO:teuthology.orchestra.run.smithi186.stdout:3244 read 0x7d3200 thru 0x7d8d54 (0x5b55 bytes) 2024-05-06T00:28:10.508 INFO:teuthology.orchestra.run.smithi186.stdout:3246 flatten 2024-05-06T00:28:10.523 INFO:teuthology.orchestra.run.smithi186.stdout:3247 write 0x8ea0400 thru 0x8ea0b0e (0x70f bytes) 2024-05-06T00:28:10.527 INFO:teuthology.orchestra.run.smithi186.stdout:3251 trunc from 0xbe39600 to 0xab98000 2024-05-06T00:28:10.532 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:10.533+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:10.549 INFO:teuthology.orchestra.run.smithi186.stdout:3254 punch from 0x6f51a00 to 0x6f5ba00, (0xa000 bytes) 2024-05-06T00:28:10.550 INFO:teuthology.orchestra.run.smithi186.stdout:3255 punch from 0x3d81200 to 0x3d8d400, (0xc200 bytes) 2024-05-06T00:28:10.550 INFO:teuthology.orchestra.run.smithi186.stdout:3257 write 0xab38600 thru 0xab3d2c5 (0x4cc6 bytes) 2024-05-06T00:28:10.551 INFO:teuthology.orchestra.run.smithi186.stdout:3258 write 0xa207600 thru 0xa213ad4 (0xc4d5 bytes) 2024-05-06T00:28:10.551 INFO:teuthology.orchestra.run.smithi186.stdout:3259 punch from 0x7141e00 to 0x7143c00, (0x1e00 bytes) 2024-05-06T00:28:10.577 INFO:teuthology.orchestra.run.smithi186.stdout:3263 write 0xc63d400 thru 0xc641cba (0x48bb bytes) 2024-05-06T00:28:10.578 INFO:teuthology.orchestra.run.smithi186.stdout:3264 read 0xae06a00 thru 0xae0d07f (0x6680 bytes) 2024-05-06T00:28:10.579 INFO:teuthology.orchestra.run.smithi186.stdout:3266 read 0xc108e00 thru 0xc112109 (0x930a bytes) 2024-05-06T00:28:10.579 INFO:teuthology.orchestra.run.smithi186.stdout:3268 write 0x2104200 thru 0x210c250 (0x8051 bytes) 2024-05-06T00:28:10.580 INFO:teuthology.orchestra.run.smithi186.stdout:3269 read 0x9cda200 thru 0x9ce3f7c (0x9d7d bytes) 2024-05-06T00:28:10.581 INFO:teuthology.orchestra.run.smithi186.stdout:3270 read 0x65a6600 thru 0x65b5c21 (0xf622 bytes) 2024-05-06T00:28:10.582 INFO:teuthology.orchestra.run.smithi186.stdout:3271 read 0x341da00 thru 0x342c155 (0xe756 bytes) 2024-05-06T00:28:10.583 INFO:teuthology.orchestra.run.smithi186.stdout:3275 read 0x877d400 thru 0x878b8f2 (0xe4f3 bytes) 2024-05-06T00:28:10.583 INFO:teuthology.orchestra.run.smithi186.stdout:3277 punch from 0x559a00 to 0x562000, (0x8600 bytes) 2024-05-06T00:28:10.584 INFO:teuthology.orchestra.run.smithi186.stdout:3278 punch from 0x9f73e00 to 0x9f83800, (0xfa00 bytes) 2024-05-06T00:28:10.584 INFO:teuthology.orchestra.run.smithi186.stdout:3282 punch from 0x6bfd600 to 0x6c0b400, (0xde00 bytes) 2024-05-06T00:28:10.585 INFO:teuthology.orchestra.run.smithi186.stdout:3283 read 0x8152800 thru 0x815ef77 (0xc778 bytes) 2024-05-06T00:28:10.586 INFO:teuthology.orchestra.run.smithi186.stdout:3284 write 0x85eb800 thru 0x85f83ca (0xcbcb bytes) 2024-05-06T00:28:10.586 INFO:teuthology.orchestra.run.smithi186.stdout:3285 read 0x1dcf000 thru 0x1ddbef6 (0xcef7 bytes) 2024-05-06T00:28:10.588 INFO:teuthology.orchestra.run.smithi186.stdout:3286 trunc from 0xc641e00 to 0xce2fc00 2024-05-06T00:28:10.614 INFO:teuthology.orchestra.run.smithi186.stdout:3290 read 0x7302c00 thru 0x730f643 (0xca44 bytes) 2024-05-06T00:28:10.615 INFO:teuthology.orchestra.run.smithi186.stdout:3293 write 0x93a3c00 thru 0x93a5fe2 (0x23e3 bytes) 2024-05-06T00:28:10.615 INFO:teuthology.orchestra.run.smithi186.stdout:3296 punch from 0x5a7be00 to 0x5a7c600, (0x800 bytes) 2024-05-06T00:28:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:10.628 INFO:teuthology.orchestra.run.smithi186.stdout:3298 trunc from 0xce2fc00 to 0x7100c00 2024-05-06T00:28:10.634 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:10.631+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:10.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:10.657 INFO:teuthology.orchestra.run.smithi186.stdout:3300 read 0x2e46200 thru 0x2e4c0e8 (0x5ee9 bytes) 2024-05-06T00:28:10.660 INFO:teuthology.orchestra.run.smithi186.stdout:3302 punch from 0x5cda200 to 0x5cdc200, (0x2000 bytes) 2024-05-06T00:28:10.660 INFO:teuthology.orchestra.run.smithi186.stdout:3303 read 0x14be200 thru 0x14ccd3e (0xeb3f bytes) 2024-05-06T00:28:10.691 INFO:teuthology.orchestra.run.smithi186.stdout:3306 write 0xec1c600 thru 0xec2a90e (0xe30f bytes) 2024-05-06T00:28:10.693 INFO:teuthology.orchestra.run.smithi186.stdout:3310 punch from 0x92f3400 to 0x92fc800, (0x9400 bytes) 2024-05-06T00:28:10.717 INFO:teuthology.orchestra.run.smithi186.stdout:3311 trunc from 0xec2aa00 to 0x2586200 2024-05-06T00:28:10.734 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:10.733+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:10.748 INFO:teuthology.orchestra.run.smithi186.stdout:3317 read 0x251e00 thru 0x260c3c (0xee3d bytes) 2024-05-06T00:28:10.751 INFO:teuthology.orchestra.run.smithi186.stdout:3319 read 0xfcc800 thru 0xfd7406 (0xac07 bytes) 2024-05-06T00:28:10.752 INFO:teuthology.orchestra.run.smithi186.stdout:3321 punch from 0x1cc4200 to 0x1ccbe00, (0x7c00 bytes) 2024-05-06T00:28:10.786 INFO:teuthology.orchestra.run.smithi186.stdout:3323 write 0xc345200 thru 0xc352abd (0xd8be bytes) 2024-05-06T00:28:10.806 INFO:teuthology.orchestra.run.smithi186.stdout:3324 trunc from 0xc352c00 to 0x138f400 2024-05-06T00:28:10.818 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:10.818 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:10.818 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:10.818 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:10.818 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:10.835 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:10.831+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:10.836 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:10.847 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:28:10.873 INFO:teuthology.orchestra.run.smithi186.stdout:3326 write 0xabfae00 thru 0xabfb6fb (0x8fc bytes) 2024-05-06T00:28:10.875 INFO:teuthology.orchestra.run.smithi186.stdout:3327 read 0x861000 thru 0x86586d (0x486e bytes) 2024-05-06T00:28:10.883 INFO:teuthology.orchestra.run.smithi186.stdout:3329 trunc from 0xabfb800 to 0xe19fe00 2024-05-06T00:28:10.902 INFO:teuthology.orchestra.run.smithi186.stdout:3330 read 0xb981200 thru 0xb98c2c0 (0xb0c1 bytes) 2024-05-06T00:28:10.905 INFO:teuthology.orchestra.run.smithi186.stdout:3333 write 0xcc14600 thru 0xcc19f82 (0x5983 bytes) 2024-05-06T00:28:10.905 INFO:teuthology.orchestra.run.smithi186.stdout:3334 read 0xab53600 thru 0xab59182 (0x5b83 bytes) 2024-05-06T00:28:10.905 INFO:teuthology.orchestra.run.smithi186.stdout:3336 write 0xaeb0400 thru 0xaeb5b3a (0x573b bytes) 2024-05-06T00:28:10.905 INFO:teuthology.orchestra.run.smithi186.stdout:3338 read 0x5c13200 thru 0x5c1cf48 (0x9d49 bytes) 2024-05-06T00:28:10.919 INFO:teuthology.orchestra.run.smithi186.stdout:3339 trunc from 0xe19fe00 to 0x611bc00 2024-05-06T00:28:10.936 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:10.933+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:10.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:10.958 INFO:teuthology.orchestra.run.smithi186.stdout:3341 read 0x2b6000 thru 0x2bab02 (0x4b03 bytes) 2024-05-06T00:28:10.978 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.971684508000003, 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:28:10.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.971455935999998, 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:28:10.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.971378651999999, 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.988 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:28:10.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:28:10.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:28:10.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:28:10.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:28:10.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:28:10.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:28:10.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:10.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:28:10.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:28:10.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.994 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:10.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:10.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.995 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:10.995 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:10.995 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.971182671000001, 2024-05-06T00:28:10.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:28:10.995 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:10.995 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:10.995 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:10.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.064073295, 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:28:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:28:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:28:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:28:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:28:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:28:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:28:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.064012416000001, 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:28:11.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.009 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:11.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:11.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9464559660000003, 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.018 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9464112379999996, 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.019 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9463728370000002, 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:11.027 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.946335575, 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:11.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:28:11.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.031 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9462986659999997, 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9462607480000003, 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:11.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9462236329999998, 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:28:11.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9404431840000003, 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:11.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:28:11.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:28:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9404050999999995, 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:11.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:28:11.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:28:11.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9403674259999999, 2024-05-06T00:28:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:28:11.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:28:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9309319160000005, 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:28:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9308994889999997, 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9308756069999999, 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.069 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.9308556770000003, 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:28:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:28:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:11.075 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:11.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:11.079 INFO:teuthology.orchestra.run.smithi186.stdout:3342 write 0xbec4c00 thru 0xbed16e3 (0xcae4 bytes) 2024-05-06T00:28:11.079 INFO:teuthology.orchestra.run.smithi186.stdout:3343 read 0x5e43a00 thru 0x5e50750 (0xcd51 bytes) 2024-05-06T00:28:11.079 INFO:teuthology.orchestra.run.smithi186.stdout:3346 trunc from 0xbed1800 to 0xe6a7a00 2024-05-06T00:28:11.079 INFO:teuthology.orchestra.run.smithi186.stdout:3347 write 0x4631a00 thru 0x4638e42 (0x7443 bytes) 2024-05-06T00:28:11.079 INFO:teuthology.orchestra.run.smithi186.stdout:3348 write 0xd2ab400 thru 0xd2ab83b (0x43c bytes) 2024-05-06T00:28:11.079 INFO:teuthology.orchestra.run.smithi186.stdout:3353 read 0xdd86200 thru 0xdd92079 (0xbe7a bytes) 2024-05-06T00:28:11.079 INFO:teuthology.orchestra.run.smithi186.stdout:3354 punch from 0x5544c00 to 0x5551400, (0xc800 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3356 write 0xaafd600 thru 0xab0d0f7 (0xfaf8 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0xedb8a00 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3358 trunc from 0xe6a7a00 to 0xedb8a00 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3360 read 0x4cde000 thru 0x4cebc31 (0xdc32 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3361 write 0x6c3fe00 thru 0x6c420da (0x22db bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3362 punch from 0x7e09e00 to 0x7e12200, (0x8400 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3363 write 0xd762a00 thru 0xd766827 (0x3e28 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3365 read 0x581f200 thru 0x582a671 (0xb472 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3370 write 0x1f5e800 thru 0x1f6172e (0x2f2f bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3371 read 0x5862800 thru 0x5867010 (0x4811 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3372 read 0xdf3c200 thru 0xdf4670d (0xa50e bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3374 trunc from 0xedb8a00 to 0xc120800 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3375 read 0x26a400 thru 0x271411 (0x7012 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3377 write 0xab9fc00 thru 0xababc76 (0xc077 bytes) 2024-05-06T00:28:11.080 INFO:teuthology.orchestra.run.smithi186.stdout:3378 read 0x5d72000 thru 0x5d74551 (0x2552 bytes) 2024-05-06T00:28:11.081 INFO:teuthology.orchestra.run.smithi186.stdout:3381 trunc from 0xc120800 to 0xb763400 2024-05-06T00:28:11.081 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:11.071+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:11.105 INFO:teuthology.orchestra.run.smithi186.stdout:3383 write 0x382e400 thru 0x3832b84 (0x4785 bytes) 2024-05-06T00:28:11.106 INFO:teuthology.orchestra.run.smithi186.stdout:3384 read 0x26cfa00 thru 0x26dd5d4 (0xdbd5 bytes) 2024-05-06T00:28:11.107 INFO:teuthology.orchestra.run.smithi186.stdout:3386 read 0x6932400 thru 0x69391ca (0x6dcb bytes) 2024-05-06T00:28:11.109 INFO:teuthology.orchestra.run.smithi186.stdout:3388 read 0x5ebcc00 thru 0x5ec161d (0x4a1e bytes) 2024-05-06T00:28:11.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:11.118 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:11.118 INFO:teuthology.orchestra.run.smithi186.stdout:3390 trunc from 0xb763400 to 0x6b99a00 2024-05-06T00:28:11.138 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:11.138 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:11.143 INFO:teuthology.orchestra.run.smithi186.stdout:3391 write 0x10c6200 thru 0x10cefed (0x8dee bytes) 2024-05-06T00:28:11.145 INFO:teuthology.orchestra.run.smithi186.stdout:3393 write 0x1507e00 thru 0x150ce9f (0x50a0 bytes) 2024-05-06T00:28:11.147 INFO:teuthology.orchestra.run.smithi186.stdout:3395 read 0x62ee000 thru 0x62f3ae0 (0x5ae1 bytes) 2024-05-06T00:28:11.147 INFO:teuthology.orchestra.run.smithi186.stdout:3396 read 0x525d200 thru 0x526accf (0xdad0 bytes) 2024-05-06T00:28:11.152 INFO:teuthology.orchestra.run.smithi186.stdout:3397 trunc from 0x6b99a00 to 0x5116000 2024-05-06T00:28:11.157 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:11.157 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:11.177 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:11.175+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:11.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:11.179 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:11.188 INFO:teuthology.orchestra.run.smithi186.stdout:3398 trunc from 0x5116000 to 0xa466a00 2024-05-06T00:28:11.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:11.203 INFO:teuthology.orchestra.run.smithi186.stdout:3399 read 0xc72600 thru 0xc7f89d (0xd29e bytes) 2024-05-06T00:28:11.216 INFO:teuthology.orchestra.run.smithi186.stdout:3400 write 0xa588c00 thru 0xa597cb2 (0xf0b3 bytes) 2024-05-06T00:28:11.217 INFO:teuthology.orchestra.run.smithi186.stdout:3401 write 0x15f5600 thru 0x15fbebb (0x68bc bytes) 2024-05-06T00:28:11.219 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:11.219 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:11.224 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:11.225 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:11.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:11.237 INFO:teuthology.orchestra.run.smithi186.stdout:3402 trunc from 0xa597e00 to 0x2701a00 2024-05-06T00:28:11.270 INFO:teuthology.orchestra.run.smithi186.stdout:3406 read 0xda0800 thru 0xdace33 (0xc634 bytes) 2024-05-06T00:28:11.285 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:11.285+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:11.298 INFO:teuthology.orchestra.run.smithi186.stdout:3407 write 0x6635800 thru 0x6640755 (0xaf56 bytes) 2024-05-06T00:28:11.300 INFO:teuthology.orchestra.run.smithi186.stdout:3409 read 0x93be00 thru 0x941cf7 (0x5ef8 bytes) 2024-05-06T00:28:11.301 INFO:teuthology.orchestra.run.smithi186.stdout:3410 punch from 0x33e2200 to 0x33f0600, (0xe400 bytes) 2024-05-06T00:28:11.337 INFO:teuthology.orchestra.run.smithi186.stdout:3412 write 0xe920a00 thru 0xe92e902 (0xdf03 bytes) 2024-05-06T00:28:11.339 INFO:teuthology.orchestra.run.smithi186.stdout:3413 read 0x635bc00 thru 0x6363d1b (0x811c bytes) 2024-05-06T00:28:11.375 INFO:teuthology.orchestra.run.smithi186.stdout:3414 trunc from 0xe92ea00 to 0x8e800 2024-05-06T00:28:11.378 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:11.377+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:11.403 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:11.404 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:11.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:11.479 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:11.477+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:11.495 INFO:teuthology.orchestra.run.smithi186.stdout:3416 write 0x452fe00 thru 0x453c7bf (0xc9c0 bytes) 2024-05-06T00:28:11.501 INFO:teuthology.orchestra.run.smithi186.stdout:3418 trunc from 0x453c800 to 0x617ce00 2024-05-06T00:28:11.533 INFO:teuthology.orchestra.run.smithi186.stdout:3419 punch from 0x5379800 to 0x537ee00, (0x5600 bytes) 2024-05-06T00:28:11.535 INFO:teuthology.orchestra.run.smithi186.stdout:3421 punch from 0x55c1a00 to 0x55cf400, (0xda00 bytes) 2024-05-06T00:28:11.535 INFO:teuthology.orchestra.run.smithi186.stdout:3423 read 0x4983000 thru 0x4984e97 (0x1e98 bytes) 2024-05-06T00:28:11.535 INFO:teuthology.orchestra.run.smithi186.stdout:3424 write 0x1abb200 thru 0x1ac8430 (0xd231 bytes) 2024-05-06T00:28:11.535 INFO:teuthology.orchestra.run.smithi186.stdout:3426 read 0x4899600 thru 0x48a1d1b (0x871c bytes) 2024-05-06T00:28:11.535 INFO:teuthology.orchestra.run.smithi186.stdout:3427 write 0x39fd600 thru 0x3a042ff (0x6d00 bytes) 2024-05-06T00:28:11.543 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:11.543 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:11.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:11.575 INFO:teuthology.orchestra.run.smithi186.stdout:3429 write 0x70a1e00 thru 0x70a6ea6 (0x50a7 bytes) 2024-05-06T00:28:11.577 INFO:teuthology.orchestra.run.smithi186.stdout:3430 punch from 0x3ea6600 to 0x3ead800, (0x7200 bytes) 2024-05-06T00:28:11.578 INFO:teuthology.orchestra.run.smithi186.stdout:3431 read 0x37d0600 thru 0x37dde00 (0xd801 bytes) 2024-05-06T00:28:11.584 INFO:teuthology.orchestra.run.smithi186.stdout:3433 trunc from 0x70a7000 to 0x4472600 2024-05-06T00:28:11.605 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:11.603+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:11.659 INFO:teuthology.orchestra.run.smithi186.stdout:3434 write 0xab6ec00 thru 0xab7dbf4 (0xeff5 bytes) 2024-05-06T00:28:11.660 INFO:teuthology.orchestra.run.smithi186.stdout:3436 read 0x40c8e00 thru 0x40cfbde (0x6ddf bytes) 2024-05-06T00:28:11.662 INFO:teuthology.orchestra.run.smithi186.stdout:3440 punch from 0x270fc00 to 0x2712c00, (0x3000 bytes) 2024-05-06T00:28:11.675 INFO:teuthology.orchestra.run.smithi186.stdout:3442 trunc from 0xab7dc00 to 0x334b800 2024-05-06T00:28:11.680 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:11.680 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:11.680 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:11.681+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:11.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:11.713 INFO:teuthology.orchestra.run.smithi186.stdout:3443 read 0x1f8d400 thru 0x1f9d1ce (0xfdcf bytes) 2024-05-06T00:28:11.742 INFO:teuthology.orchestra.run.smithi186.stdout:3445 write 0x3d60000 thru 0x3d6b9df (0xb9e0 bytes) 2024-05-06T00:28:11.744 INFO:teuthology.orchestra.run.smithi186.stdout:3447 punch from 0x22d8400 to 0x22dca00, (0x4600 bytes) 2024-05-06T00:28:11.785 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:11.783+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:11.789 INFO:teuthology.orchestra.run.smithi186.stdout:3448 write 0xe4ac600 thru 0xe4af592 (0x2f93 bytes) 2024-05-06T00:28:11.791 INFO:teuthology.orchestra.run.smithi186.stdout:3449 write 0x412ee00 thru 0x413c317 (0xd518 bytes) 2024-05-06T00:28:11.791 INFO:teuthology.orchestra.run.smithi186.stdout:3450 punch from 0x6c5bc00 to 0x6c6b000, (0xf400 bytes) 2024-05-06T00:28:11.792 INFO:teuthology.orchestra.run.smithi186.stdout:3451 write 0xd3a4000 thru 0xd3ac32e (0x832f bytes) 2024-05-06T00:28:11.792 INFO:teuthology.orchestra.run.smithi186.stdout:3453 read 0x82c2c00 thru 0x82d2956 (0xfd57 bytes) 2024-05-06T00:28:11.792 INFO:teuthology.orchestra.run.smithi186.stdout:3455 write 0x1907e00 thru 0x1909c6f (0x1e70 bytes) 2024-05-06T00:28:11.792 INFO:teuthology.orchestra.run.smithi186.stdout:3457 write 0xdd18a00 thru 0xdd1a9e0 (0x1fe1 bytes) 2024-05-06T00:28:11.818 INFO:teuthology.orchestra.run.smithi186.stdout:3459 trunc from 0xe4af600 to 0x160c00 2024-05-06T00:28:11.861 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:11.862 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:11.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:11.880 INFO:teuthology.orchestra.run.smithi186.stdout:3460 read 0x8c600 thru 0x9960a (0xd00b bytes) 2024-05-06T00:28:11.883 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:11.881+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:11.906 INFO:teuthology.orchestra.run.smithi186.stdout:3461 trunc from 0x160c00 to 0xcf84a00 2024-05-06T00:28:11.926 INFO:teuthology.orchestra.run.smithi186.stdout:3463 read 0x48f400 thru 0x49dbb6 (0xe7b7 bytes) 2024-05-06T00:28:11.941 INFO:teuthology.orchestra.run.smithi186.stdout:3464 trunc from 0xcf84a00 to 0x8bb3800 2024-05-06T00:28:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:3470 trunc from 0x8bb3800 to 0xb982000 2024-05-06T00:28:11.984 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:11.979+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:11.999 INFO:teuthology.orchestra.run.smithi186.stdout:3471 read 0x7c70600 thru 0x7c7e48a (0xde8b bytes) 2024-05-06T00:28:12.000 INFO:teuthology.orchestra.run.smithi186.stdout:3472 write 0x642b800 thru 0x6437b11 (0xc312 bytes) 2024-05-06T00:28:12.001 INFO:teuthology.orchestra.run.smithi186.stdout:3473 read 0xb111c00 thru 0xb1133fe (0x17ff bytes) 2024-05-06T00:28:12.001 INFO:teuthology.orchestra.run.smithi186.stdout:3475 write 0xa4e7400 thru 0xa4f0c5f (0x9860 bytes) 2024-05-06T00:28:12.001 INFO:teuthology.orchestra.run.smithi186.stdout:3476 write 0x24cb800 thru 0x24d2f32 (0x7733 bytes) 2024-05-06T00:28:12.001 INFO:teuthology.orchestra.run.smithi186.stdout:3477 punch from 0x6afb400 to 0x6aff000, (0x3c00 bytes) 2024-05-06T00:28:12.002 INFO:teuthology.orchestra.run.smithi186.stdout:3479 trunc from 0xb982000 to 0xbf69c00 2024-05-06T00:28:12.040 INFO:teuthology.orchestra.run.smithi186.stdout:3480 read 0x60ac000 thru 0x60b37f5 (0x77f6 bytes) 2024-05-06T00:28:12.041 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:12.041 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:12.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:12.057 INFO:teuthology.orchestra.run.smithi186.stdout:3482 trunc from 0xbf69c00 to 0x292aa00 2024-05-06T00:28:12.085 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:12.085+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:12.101 INFO:teuthology.orchestra.run.smithi186.stdout:3485 write 0x26db800 thru 0x26e7180 (0xb981 bytes) 2024-05-06T00:28:12.103 INFO:teuthology.orchestra.run.smithi186.stdout:skipping zero length punch hole 2024-05-06T00:28:12.103 INFO:teuthology.orchestra.run.smithi186.stdout:3488 punch from 0x37da00 to 0x385600, (0x7c00 bytes) 2024-05-06T00:28:12.107 INFO:teuthology.orchestra.run.smithi186.stdout:3492 trunc from 0x292aa00 to 0x45d8a00 2024-05-06T00:28:12.135 INFO:teuthology.orchestra.run.smithi186.stdout:3494 read 0x2927800 thru 0x292cc53 (0x5454 bytes) 2024-05-06T00:28:12.137 INFO:teuthology.orchestra.run.smithi186.stdout:3496 read 0x391a400 thru 0x391cf2b (0x2b2c bytes) 2024-05-06T00:28:12.168 INFO:teuthology.orchestra.run.smithi186.stdout:3498 write 0x761bc00 thru 0x761ca19 (0xe1a bytes) 2024-05-06T00:28:12.186 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:12.185+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:12.215 INFO:teuthology.orchestra.run.smithi186.stdout:3500 write 0xb774800 thru 0xb774b6f (0x370 bytes) 2024-05-06T00:28:12.217 INFO:teuthology.orchestra.run.smithi186.stdout:3502 read 0xa3d7200 thru 0xa3df079 (0x7e7a bytes) 2024-05-06T00:28:12.217 INFO:teuthology.orchestra.run.smithi186.stdout:3505 read 0x19abe00 thru 0x19b5d89 (0x9f8a bytes) 2024-05-06T00:28:12.218 INFO:teuthology.orchestra.run.smithi186.stdout:3507 read 0x64af600 thru 0x64b7d7b (0x877c bytes) 2024-05-06T00:28:12.228 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:12.228 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:12.229 INFO:teuthology.orchestra.run.smithi186.stdout:3510 trunc from 0xb774c00 to 0x7846a00 2024-05-06T00:28:12.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:12.273 INFO:teuthology.orchestra.run.smithi186.stdout:3511 write 0x2522a00 thru 0x2532567 (0xfb68 bytes) 2024-05-06T00:28:12.279 INFO:teuthology.orchestra.run.smithi186.stdout:3512 read 0x24d8600 thru 0x24da444 (0x1e45 bytes) 2024-05-06T00:28:12.282 INFO:teuthology.orchestra.run.smithi186.stdout:3514 trunc from 0x7846a00 to 0x6ebf600 2024-05-06T00:28:12.289 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:12.287+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:12.348 INFO:teuthology.orchestra.run.smithi186.stdout:3520 write 0xa065c00 thru 0xa066ea6 (0x12a7 bytes) 2024-05-06T00:28:12.349 INFO:teuthology.orchestra.run.smithi186.stdout:3523 punch from 0x43ce000 to 0x43d1000, (0x3000 bytes) 2024-05-06T00:28:12.350 INFO:teuthology.orchestra.run.smithi186.stdout:3524 read 0x2d23000 thru 0x2d2a3f2 (0x73f3 bytes) 2024-05-06T00:28:12.351 INFO:teuthology.orchestra.run.smithi186.stdout:3526 read 0x40ad600 thru 0x40b9ad5 (0xc4d6 bytes) 2024-05-06T00:28:12.365 INFO:teuthology.orchestra.run.smithi186.stdout:3529 trunc from 0xa067000 to 0x301c400 2024-05-06T00:28:12.370 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:12.370 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:12.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:12.388 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:12.385+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:12.455 INFO:teuthology.orchestra.run.smithi186.stdout:3530 write 0xdf5e200 thru 0xdf695a3 (0xb3a4 bytes) 2024-05-06T00:28:12.457 INFO:teuthology.orchestra.run.smithi186.stdout:3533 read 0xa22e400 thru 0xa231d88 (0x3989 bytes) 2024-05-06T00:28:12.457 INFO:teuthology.orchestra.run.smithi186.stdout:3536 punch from 0xdae8200 to 0xdaed200, (0x5000 bytes) 2024-05-06T00:28:12.491 INFO:teuthology.orchestra.run.smithi186.stdout:3539 write 0xe5d4000 thru 0xe5e188a (0xd88b bytes) 2024-05-06T00:28:12.493 INFO:teuthology.orchestra.run.smithi186.stdout:3541 write 0x362ce00 thru 0x3633d5b (0x6f5c bytes) 2024-05-06T00:28:12.500 INFO:teuthology.orchestra.run.smithi186.stdout:3542 trunc from 0xe5e1a00 to 0xb07a200 2024-05-06T00:28:12.516 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:12.487+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:12.554 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:12.554 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:12.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:12.587 INFO:teuthology.orchestra.run.smithi186.stdout:3545 write 0xd022a00 thru 0xd03279a (0xfd9b bytes) 2024-05-06T00:28:12.590 INFO:teuthology.orchestra.run.smithi186.stdout:3546 read 0x6586600 thru 0x658c9da (0x63db bytes) 2024-05-06T00:28:12.590 INFO:teuthology.orchestra.run.smithi186.stdout:3549 write 0x32c6200 thru 0x32c9531 (0x3332 bytes) 2024-05-06T00:28:12.590 INFO:teuthology.orchestra.run.smithi186.stdout:3552 read 0x98ed600 thru 0x98ee500 (0xf01 bytes) 2024-05-06T00:28:12.590 INFO:teuthology.orchestra.run.smithi186.stdout:3557 read 0x7082000 thru 0x7087466 (0x5467 bytes) 2024-05-06T00:28:12.590 INFO:teuthology.orchestra.run.smithi186.stdout:3558 read 0x6c0e400 thru 0x6c1b775 (0xd376 bytes) 2024-05-06T00:28:12.591 INFO:teuthology.orchestra.run.smithi186.stdout:3560 write 0x679cc00 thru 0x67a7083 (0xa484 bytes) 2024-05-06T00:28:12.591 INFO:teuthology.orchestra.run.smithi186.stdout:3562 read 0x78bb400 thru 0x78c50fd (0x9cfe bytes) 2024-05-06T00:28:12.609 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:12.607+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:12.614 INFO:teuthology.orchestra.run.smithi186.stdout:3565 trunc from 0xd032800 to 0x2ed4000 2024-05-06T00:28:12.691 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:12.687+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:12.695 INFO:teuthology.orchestra.run.smithi186.stdout:3568 write 0xaf93600 thru 0xafa2f84 (0xf985 bytes) 2024-05-06T00:28:12.696 INFO:teuthology.orchestra.run.smithi186.stdout:3569 read 0x5fca800 thru 0x5fd578a (0xaf8b bytes) 2024-05-06T00:28:12.697 INFO:teuthology.orchestra.run.smithi186.stdout:3570 read 0x6633c00 thru 0x663ca9b (0x8e9c bytes) 2024-05-06T00:28:12.697 INFO:teuthology.orchestra.run.smithi186.stdout:3571 punch from 0x3de0400 to 0x3de5400, (0x5000 bytes) 2024-05-06T00:28:12.697 INFO:teuthology.orchestra.run.smithi186.stdout:3576 write 0xf92200 thru 0xf92c29 (0xa2a bytes) 2024-05-06T00:28:12.699 INFO:teuthology.orchestra.run.smithi186.stdout:3577 read 0xb06a00 thru 0xb08d05 (0x2306 bytes) 2024-05-06T00:28:12.701 INFO:teuthology.orchestra.run.smithi186.stdout:3579 read 0x8e95600 thru 0x8ea02b3 (0xacb4 bytes) 2024-05-06T00:28:12.701 INFO:teuthology.orchestra.run.smithi186.stdout:3580 write 0xad7400 thru 0xad76b9 (0x2ba bytes) 2024-05-06T00:28:12.701 INFO:teuthology.orchestra.run.smithi186.stdout:3582 punch from 0x8f11c00 to 0x8f1c800, (0xac00 bytes) 2024-05-06T00:28:12.702 INFO:teuthology.orchestra.run.smithi186.stdout:3583 write 0x2b29c00 thru 0x2b36fef (0xd3f0 bytes) 2024-05-06T00:28:12.703 INFO:teuthology.orchestra.run.smithi186.stdout:3584 read 0xa131000 thru 0xa13f69f (0xe6a0 bytes) 2024-05-06T00:28:12.703 INFO:teuthology.orchestra.run.smithi186.stdout:3587 read 0x7a13a00 thru 0x7a1e804 (0xae05 bytes) 2024-05-06T00:28:12.712 INFO:teuthology.orchestra.run.smithi186.stdout:3589 trunc from 0xafa3000 to 0x689c000 2024-05-06T00:28:12.717 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:12.717 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:12.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:12.742 INFO:teuthology.orchestra.run.smithi186.stdout:3590 read 0xee2800 thru 0xeef5ca (0xcdcb bytes) 2024-05-06T00:28:12.745 INFO:teuthology.orchestra.run.smithi186.stdout:3591 read 0x4a36000 thru 0x4a3f425 (0x9426 bytes) 2024-05-06T00:28:12.745 INFO:teuthology.orchestra.run.smithi186.stdout:3595 read 0x1fbc400 thru 0x1fc4fb2 (0x8bb3 bytes) 2024-05-06T00:28:12.755 INFO:teuthology.orchestra.run.smithi186.stdout:3597 trunc from 0x689c000 to 0x45b9a00 2024-05-06T00:28:12.779 INFO:teuthology.orchestra.run.smithi186.stdout:3599 read 0x22c2a00 thru 0x22d293c (0xff3d bytes) 2024-05-06T00:28:12.792 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:12.793+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:12.813 INFO:teuthology.orchestra.run.smithi186.stdout:3600 write 0xafb6200 thru 0xafbeada (0x88db bytes) 2024-05-06T00:28:12.842 INFO:teuthology.orchestra.run.smithi186.stdout:3603 write 0xc2ca600 thru 0xc2d0791 (0x6192 bytes) 2024-05-06T00:28:12.843 INFO:teuthology.orchestra.run.smithi186.stdout:3605 write 0x7280200 thru 0x7288901 (0x8702 bytes) 2024-05-06T00:28:12.845 INFO:teuthology.orchestra.run.smithi186.stdout:3606 trunc from 0xc2d0800 to 0xcbfc400 2024-05-06T00:28:12.863 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:12.863 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:12.872 INFO:teuthology.orchestra.run.smithi186.stdout:3608 write 0x40b7200 thru 0x40bd6bb (0x64bc bytes) 2024-05-06T00:28:12.874 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:28:12.874 INFO:teuthology.orchestra.run.smithi186.stdout:3609 write 0x7162e00 thru 0x716a8cf (0x7ad0 bytes) 2024-05-06T00:28:12.875 INFO:teuthology.orchestra.run.smithi186.stdout:3610 read 0x5863400 thru 0x5864c8d (0x188e bytes) 2024-05-06T00:28:12.877 INFO:teuthology.orchestra.run.smithi186.stdout:3612 trunc from 0xcbfc400 to 0xdadd800 2024-05-06T00:28:12.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:12.893 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:12.891+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:12.897 INFO:teuthology.orchestra.run.smithi186.stdout:3613 read 0x7769200 thru 0x777221b (0x901c bytes) 2024-05-06T00:28:12.900 INFO:teuthology.orchestra.run.smithi186.stdout:3614 read 0x65bc600 thru 0x65bf29b (0x2c9c bytes) 2024-05-06T00:28:12.903 INFO:teuthology.orchestra.run.smithi186.stdout:3615 trunc from 0xdadd800 to 0xcb01200 2024-05-06T00:28:12.918 INFO:teuthology.orchestra.run.smithi186.stdout:3616 write 0xbcc0600 thru 0xbcc4255 (0x3c56 bytes) 2024-05-06T00:28:12.920 INFO:teuthology.orchestra.run.smithi186.stdout:3617 write 0x844f400 thru 0x845a375 (0xaf76 bytes) 2024-05-06T00:28:12.920 INFO:teuthology.orchestra.run.smithi186.stdout:3618 read 0x4cc4a00 thru 0x4cc698b (0x1f8c bytes) 2024-05-06T00:28:12.920 INFO:teuthology.orchestra.run.smithi186.stdout:3621 read 0x3d38600 thru 0x3d3e84f (0x6250 bytes) 2024-05-06T00:28:12.921 INFO:teuthology.orchestra.run.smithi186.stdout:3622 clone 24 order 22 su 262144 sc 3 2024-05-06T00:28:12.994 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:12.991+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:13.043 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:13.043 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:13.058 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:13.095 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:13.091+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:13.175 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:13.175 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:13.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:13.196 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:13.195+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:13.241 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:13.296 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:13.295+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:13.310 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:13.310 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:13.326 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:13.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:13.382 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:13.397 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:13.398 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:13.395+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:13.398 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:13.414 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:13.414 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:13.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:13.435 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:13.447 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:13.447 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:13.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:13.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:13.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21566 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 417792~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4202727270000004, 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090303018999999998, 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21566 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5400000000000002e-07 2024-05-06T00:28:13.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026960+0000", 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4449999999999999e-06 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026966+0000", 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6200000000000001e-06 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026978+0000", 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1248e-05 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116885+0000", 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.089907163999999998 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116986+0000", 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101363 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117255+0000", 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026882499999999999 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21568 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 423936~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4201652419999995, 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090403081999999996, 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21568 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027060+0000", 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027063+0000", 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3249999999999999e-06 2024-05-06T00:28:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027072+0000", 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2859999999999999e-06 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027078+0000", 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117286+0000", 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090207976999999995 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117372+0000", 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5865000000000005e-05 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117463+0000", 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0662999999999996e-05 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21570 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 430080~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4201386330000005, 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090545366000000002, 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21570 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027087+0000", 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3799999999999998e-07 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027089+0000", 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.12e-06 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027095+0000", 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4590000000000003e-06 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027100+0000", 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117492+0000", 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090391632999999999 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117546+0000", 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4177e-05 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117631+0000", 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5173999999999995e-05 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21572 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 436224~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4201162719999996, 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090693918999999998, 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21572 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027109+0000", 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7100000000000002e-07 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027112+0000", 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5490000000000001e-06 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027119+0000", 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7829999999999999e-06 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027124+0000", 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.04e-06 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117662+0000", 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090537113000000002 2024-05-06T00:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117717+0000", 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5798999999999999e-05 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117802+0000", 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5063999999999997e-05 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4200922059999996, 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:13.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:28:13.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:28:13.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4200714170000008, 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.470 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:28:13.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:28:13.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.420051355, 2024-05-06T00:28:13.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:13.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.478 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4200320790000003, 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4200119610000002, 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4199805520000002, 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:28:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4199476400000002, 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:28:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:28:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4199270530000003, 2024-05-06T00:28:13.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:28:13.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:28:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4199076730000009, 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:28:13.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4198629920000005, 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.505 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.509 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4197979010000008, 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4197663269999996, 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.514 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:13.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:28:13.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4197440659999998, 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:28:13.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:28:13.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4197229500000006, 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.523 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:13.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7620063239999997, 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:13.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:28:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:28:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7615960820000005, 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:28:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:13.536 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:13.537 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:13.539 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:13.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:13.541 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:13.535+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:13.639 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:13.637+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:13.710 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:13.710 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:13.710 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:13.710 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:13.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:13.769 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:13.769+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:13.841 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:13.841+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:13.868 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:28:13.878 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:13.878 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:13.878 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [1, 2, 3, 4, 0] 2024-05-06T00:28:13.879 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-05-06T00:28:13.901 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:13.902 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:13.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:13.961 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:13.961+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:14.042 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:14.038+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:14.079 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.071947014999999, 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:28:14.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:28:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.071718443000002, 2024-05-06T00:28:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:28:14.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:28:14.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.071641159000002, 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:28:14.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:28:14.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.071445177999998, 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:28:14.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:28:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.164335802, 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.098 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:28:14.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.164274923000001, 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:28:14.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.046718473, 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:28:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.046673745, 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:28:14.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.115 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.046635344, 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.046598081999999, 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:28:14.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:28:14.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.046561173000001, 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.046523255, 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:14.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:28:14.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:28:14.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:28:14.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.046486140000001, 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:14.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:28:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.040705690999999, 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:28:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:28:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.040667607, 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.040629933, 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:28:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.031194423000001, 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.031161996, 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.031138114000001, 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:28:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.031118184, 2024-05-06T00:28:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:28:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:28:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:14.166 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:14.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:14.168 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:14.166+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:14.186 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-06T00:28:14.185+0000 7fdfb36a9640 -1 mon.a@0(leader).osd e89 definitely_dead 0 2024-05-06T00:28:14.269 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:14.266+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:14.299 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:14.299 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:14.299 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:14.299 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:14.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:14.405 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:14.402+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:14.471 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:14.469+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:14.488 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:14.489 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:14.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:14.518 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-05-06T00:28:14.582 INFO:teuthology.orchestra.run.smithi186.stdout:3623 punch from 0xad38e00 to 0xad45600, (0xc800 bytes) 2024-05-06T00:28:14.595 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:14.590+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:14.600 INFO:teuthology.orchestra.run.smithi186.stdout:3624 write 0x466e400 thru 0x467979c (0xb39d bytes) 2024-05-06T00:28:14.601 INFO:teuthology.orchestra.run.smithi186.stdout:3625 read 0x9e79e00 thru 0x9e7e04e (0x424f bytes) 2024-05-06T00:28:14.601 INFO:teuthology.orchestra.run.smithi186.stdout:3626 write 0x1548e00 thru 0x15509e1 (0x7be2 bytes) 2024-05-06T00:28:14.602 INFO:teuthology.orchestra.run.smithi186.stdout:3627 write 0x9664400 thru 0x966d023 (0x8c24 bytes) 2024-05-06T00:28:14.602 INFO:teuthology.orchestra.run.smithi186.stdout:3628 read 0xa35d800 thru 0xa3625dd (0x4dde bytes) 2024-05-06T00:28:14.603 INFO:teuthology.orchestra.run.smithi186.stdout:3630 write 0x2874c00 thru 0x2882cfd (0xe0fe bytes) 2024-05-06T00:28:14.604 INFO:teuthology.orchestra.run.smithi186.stdout:3632 punch from 0x306b000 to 0x3074a00, (0x9a00 bytes) 2024-05-06T00:28:14.638 INFO:teuthology.orchestra.run.smithi186.stdout:3633 trunc from 0xcb01200 to 0xba8a600 2024-05-06T00:28:14.674 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 105.070779703, 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.675 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:14.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.798131003999998, 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:28:14.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.798095828000001, 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.688 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:28:14.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.798007149, 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.694 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:28:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.797905206000003, 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:28:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.797796104, 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:28:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:28:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:28:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.797698244999999, 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.797633195000003, 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:28:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:28:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.797540548999997, 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:28:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:28:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:28:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.797468545000001, 2024-05-06T00:28:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:28:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:28:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.038007059000002, 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.735 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.740 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.037987811000001, 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:28:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:28:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.037937691000003, 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:28:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.037916639999999, 2024-05-06T00:28:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:28:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.037901321, 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.037884439000003, 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.769 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.037866102999999, 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:28:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.037850100999997, 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:28:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.037763738000002, 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:28:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.027801261999997, 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:14.792 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:14.793 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:14.670+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:14.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:14.798 INFO:teuthology.orchestra.run.smithi186.stdout:3634 trunc from 0xba8a600 to 0xd538e00 2024-05-06T00:28:14.798 INFO:teuthology.orchestra.run.smithi186.stdout:3635 read 0x8935000 thru 0x893e98e (0x998f bytes) 2024-05-06T00:28:14.798 INFO:teuthology.orchestra.run.smithi186.stdout:3636 write 0x94aa400 thru 0x94b2468 (0x8069 bytes) 2024-05-06T00:28:14.798 INFO:teuthology.orchestra.run.smithi186.stdout:3637 write 0x9f53c00 thru 0x9f55884 (0x1c85 bytes) 2024-05-06T00:28:14.798 INFO:teuthology.orchestra.run.smithi186.stdout:3638 trunc from 0xd538e00 to 0xa04e000 2024-05-06T00:28:14.799 INFO:teuthology.orchestra.run.smithi186.stdout:3640 read 0x9ea0200 thru 0x9ea497c (0x477d bytes) 2024-05-06T00:28:14.799 INFO:teuthology.orchestra.run.smithi186.stdout:3641 read 0x7b45c00 thru 0x7b4b179 (0x557a bytes) 2024-05-06T00:28:14.799 INFO:teuthology.orchestra.run.smithi186.stdout:3643 read 0x1014800 thru 0x10187dc (0x3fdd bytes) 2024-05-06T00:28:14.799 INFO:teuthology.orchestra.run.smithi186.stdout:3644 punch from 0x1ee9800 to 0x1ef2a00, (0x9200 bytes) 2024-05-06T00:28:14.799 INFO:teuthology.orchestra.run.smithi186.stdout:3646 punch from 0x3f64200 to 0x3f71a00, (0xd800 bytes) 2024-05-06T00:28:14.799 INFO:teuthology.orchestra.run.smithi186.stdout:3647 write 0x1c77000 thru 0x1c7b91d (0x491e bytes) 2024-05-06T00:28:14.799 INFO:teuthology.orchestra.run.smithi186.stdout:3648 read 0x696e000 thru 0x69765cf (0x85d0 bytes) 2024-05-06T00:28:14.799 INFO:teuthology.orchestra.run.smithi186.stdout:3649 write 0x449c800 thru 0x44a362a (0x6e2b bytes) 2024-05-06T00:28:14.837 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:14.834+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:14.861 INFO:teuthology.orchestra.run.smithi186.stdout:3650 trunc from 0xa04e000 to 0x157a800 2024-05-06T00:28:14.894 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.3. 2024-05-06T00:28:14.895 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:14.890+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:14.974 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:14.974 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:14.974 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:14.974 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:14.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:15.029 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:15.026+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:15.097 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:15.094+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:15.151 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:15.151 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:15.151 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:15.151 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:15.151 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:15.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:15.209 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:15.206+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:15.299 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:15.297+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:15.333 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:15.333 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:15.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.671505424999999, 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:28:15.334 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:28:15.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:15.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.338 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.69226392, 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.339 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:28:15.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.692189701, 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.343 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.347 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.692127802, 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.692068203, 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:28:15.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:28:15.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691984068, 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:28:15.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691918555000001, 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.361 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:28:15.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691860693000001, 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:28:15.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691764098, 2024-05-06T00:28:15.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:28:15.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691686373, 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:28:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:28:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691613586999999, 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:28:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691536096, 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:28:15.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691441966999999, 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.388 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:28:15.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:28:15.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691359005000001, 2024-05-06T00:28:15.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:28:15.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:28:15.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691284899999999, 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:28:15.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:28:15.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:28:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691185710999999, 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:28:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691122559, 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.691022294, 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:28:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:28:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.690932922, 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:28:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:28:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.445074239, 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:15.427 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:15.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:15.429 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:15.422+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:15.529 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:15.526+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:15.541 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:15.599 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:15.600 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:15.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:15.630 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:15.629+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:15.695 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:15.695 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:15.711 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:15.712 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:15.729 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:15.730 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:15.731 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:15.726+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:15.747 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:15.747 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:15.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:15.768 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:15.784 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:15.784 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:15.784 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:15.784 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:15.784 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:15.789 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:15.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:15.841 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:15.838+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:15.898 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:28:15.934 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:15.930+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:15.964 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:15.965 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:15.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:16.034 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:16.033+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:16.121 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:16.121 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:16.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:16.136 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:16.134+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:16.236 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:16.237+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:16.305 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:16.306 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:16.319 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:16.337 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:16.334+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:16.439 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:16.437+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:16.484 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:16.485 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:16.498 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:16.540 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:16.534+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:16.643 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:16.638+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:16.664 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:16.664 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:16.678 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:16.750 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:16.750+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:16.819 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:16.821 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:16.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:16.874 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:16.873+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:28:16.947 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:16.945+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:16.988 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:16.988 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:17.008 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:17.049 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:17.042+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:17.150 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:17.146+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:17.203 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:17.203 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:17.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:17.255 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:17.246+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:17.354 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:17.350+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:17.358 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:17.358 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:17.374 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:17.454 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:17.453+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:17.518 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:17.518 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:17.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:17.581 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:17.578+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:17.656 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:17.650+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:17.674 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:17.674 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:17.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:17.750 INFO:teuthology.orchestra.run.smithi186.stdout:3652 write 0x626ec00 thru 0x6276fae (0x83af bytes) 2024-05-06T00:28:17.781 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:17.778+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:17.794 INFO:teuthology.orchestra.run.smithi186.stdout:3653 write 0xbb15200 thru 0xbb1ddba (0x8bbb bytes) 2024-05-06T00:28:17.796 INFO:teuthology.orchestra.run.smithi186.stdout:3655 read 0x80ee200 thru 0x80ef84f (0x1650 bytes) 2024-05-06T00:28:17.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:17.835 INFO:teuthology.orchestra.run.smithi186.stdout:3661 trunc from 0xbb1de00 to 0x644a00 2024-05-06T00:28:17.855 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:17.855 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:17.858 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:17.854+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:17.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:17.878 INFO:teuthology.orchestra.run.smithi186.stdout:3662 punch from 0x5c5800 to 0x5cf800, (0xa000 bytes) 2024-05-06T00:28:17.880 INFO:teuthology.orchestra.run.smithi186.stdout:3663 read 0x590200 thru 0x59f42e (0xf22f bytes) 2024-05-06T00:28:17.959 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:17.954+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:17.974 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:17.974 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:17.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:18.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:18.009 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:18.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:18.029 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 108.436872272, 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:28:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:18.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:28:18.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:28:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:28:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:28:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:28:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.164223573000001, 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:28:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:28:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.164188396999997, 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:28:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:28:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:28:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.164099718000003, 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.163997774999999, 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:28:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:28:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.163888673000002, 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:28:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.163790814000002, 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.078 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:28:18.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.163725763999999, 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:28:18.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:28:18.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.163633118, 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.090 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:28:18.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:28:18.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:28:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:18.095 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.163561113999997, 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:18.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:28:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.404099627999997, 2024-05-06T00:28:18.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:28:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:28:18.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:28:18.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.404080380000003, 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.107 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.112 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.404030259999999, 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.113 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.118 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.404009209000002, 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:28:18.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.403993890000002, 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:28:18.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.403977007999998, 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.403958672000002, 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.136 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:28:18.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:28:18.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:28:18.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.403942669999999, 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.142 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.147 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.403856306999998, 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:18.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.393893831, 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:18.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:18.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:18.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.159 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.159 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:18.160 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:18.160 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:18.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:18.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:18.162 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:18.154+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:18.162 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:18.261 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:18.261+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:18.333 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:18.333 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:18.333 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:18.333 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:18.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:18.397 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:18.394+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:18.464 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:18.461+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:18.472 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:18.472 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:18.472 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:18.473 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:18.473 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:18.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:18.566 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:18.565+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:18.656 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:18.656 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:18.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:18.656 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:18.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.656 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:28:18.656 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:18.656 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.993766235999999, 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.657 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:18.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:28:18.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:28:18.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:28:18.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:28:18.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:28:18.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:18.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:28:18.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:28:18.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:28:18.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:28:18.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:28:18.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:28:18.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.014524731000002, 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.665 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.666 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:28:18.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.666 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:18.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:18.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:28:18.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:28:18.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:28:18.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.671 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.014450512, 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:28:18.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:28:18.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.014388613000001, 2024-05-06T00:28:18.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:28:18.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:28:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:28:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:28:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.681 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.014329014000001, 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:28:18.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.686 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:28:18.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.014244879, 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.691 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:28:18.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:28:18.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:28:18.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:28:18.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.014179366, 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.697 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.698 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:28:18.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.699 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:18.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:18.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:28:18.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:28:18.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:28:18.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:28:18.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:28:18.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:28:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:28:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:28:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:28:18.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:28:18.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:28:18.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:28:18.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:28:18.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:28:18.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.706 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.707 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.707 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:18.707 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.014121503999998, 2024-05-06T00:28:18.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:28:18.707 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.708 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.708 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.708 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:28:18.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.709 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:18.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:18.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:28:18.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:28:18.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:28:18.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:28:18.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:28:18.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:28:18.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:28:18.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:28:18.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:28:18.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:28:18.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:28:18.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:28:18.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:28:18.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:28:18.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.717 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.717 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.014024909, 2024-05-06T00:28:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:28:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:28:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:18.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:28:18.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:28:18.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:28:18.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:28:18.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:28:18.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:28:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:28:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:28:18.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:28:18.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:28:18.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:28:18.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.726 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.726 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:18.727 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013947183999999, 2024-05-06T00:28:18.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:28:18.727 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.728 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:28:18.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.728 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:28:18.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:28:18.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:28:18.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:28:18.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:28:18.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:28:18.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:28:18.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:28:18.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:28:18.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:28:18.734 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.734 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.734 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.734 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.735 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:18.735 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013874397999999, 2024-05-06T00:28:18.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:28:18.735 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.735 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.735 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.735 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.735 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:28:18.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013796907, 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:28:18.740 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:28:18.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:28:18.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013702777999999, 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.745 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.749 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013619815999999, 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:28:18.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:28:18.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013545710999999, 2024-05-06T00:28:18.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:28:18.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:28:18.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:18.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:28:18.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:28:18.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:28:18.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:28:18.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:28:18.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:28:18.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:28:18.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:28:18.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:28:18.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:28:18.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.761 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.761 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.762 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:18.762 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013446521999999, 2024-05-06T00:28:18.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:28:18.762 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.762 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.762 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.762 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.762 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.763 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.763 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:28:18.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.763 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:18.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:18.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:28:18.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:28:18.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:28:18.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.768 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.01338337, 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.769 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:28:18.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.770 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:28:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:28:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:28:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:28:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:28:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:28:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:28:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:28:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:28:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:28:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:28:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013283104999999, 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:28:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:28:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:28:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:28:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:28:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:28:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.013193733000001, 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:28:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.76733505, 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:28:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:18.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:18.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:18.793 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:18.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:18.795 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:18.793+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:18.895 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:18.893+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:18.932 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:18.932 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:18.932 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:18.932 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:18.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:18.951 INFO:teuthology.orchestra.run.smithi186.stdout:3667 write 0x96a6600 thru 0x96b23c6 (0xbdc7 bytes) 2024-05-06T00:28:18.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:28:18.966 INFO:teuthology.orchestra.run.smithi186.stdout:3668 trunc from 0x96b2400 to 0x472b400 2024-05-06T00:28:18.995 INFO:teuthology.orchestra.run.smithi186.stdout:3669 write 0xe56800 thru 0xe56c37 (0x438 bytes) 2024-05-06T00:28:18.995 INFO:teuthology.orchestra.run.smithi186.stdout:3671 read 0x14b0e00 thru 0x14be039 (0xd23a bytes) 2024-05-06T00:28:18.995 INFO:teuthology.orchestra.run.smithi186.stdout:3672 read 0xf94c00 thru 0xfa1786 (0xcb87 bytes) 2024-05-06T00:28:18.997 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:18.994+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:19.018 INFO:teuthology.orchestra.run.smithi186.stdout:3673 write 0x5ef1600 thru 0x5ef93da (0x7ddb bytes) 2024-05-06T00:28:19.049 INFO:teuthology.orchestra.run.smithi186.stdout:3674 write 0x9973600 thru 0x997af86 (0x7987 bytes) 2024-05-06T00:28:19.062 INFO:teuthology.orchestra.run.smithi186.stdout:3675 trunc from 0x997b000 to 0x3b6c800 2024-05-06T00:28:19.076 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:19.076 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:19.076 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:19.076 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:19.076 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:19.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:19.109 INFO:teuthology.orchestra.run.smithi186.stdout:3677 write 0x5764e00 thru 0x576a57f (0x5780 bytes) 2024-05-06T00:28:19.134 INFO:teuthology.orchestra.run.smithi186.stdout:3678 write 0x8cf4200 thru 0x8cf66d0 (0x24d1 bytes) 2024-05-06T00:28:19.135 INFO:teuthology.orchestra.run.smithi186.stdout:3679 write 0x4f81400 thru 0x4f87c4c (0x684d bytes) 2024-05-06T00:28:19.136 INFO:teuthology.orchestra.run.smithi186.stdout:3680 read 0x202b400 thru 0x2035f2b (0xab2c bytes) 2024-05-06T00:28:19.136 INFO:teuthology.orchestra.run.smithi186.stdout:3682 write 0x51bb000 thru 0x51c7548 (0xc549 bytes) 2024-05-06T00:28:19.136 INFO:teuthology.orchestra.run.smithi186.stdout:3683 read 0x63ee800 thru 0x63fe295 (0xfa96 bytes) 2024-05-06T00:28:19.137 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:19.134+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:19.151 INFO:teuthology.orchestra.run.smithi186.stdout:3684 trunc from 0x8cf6800 to 0x230600 2024-05-06T00:28:19.180 INFO:teuthology.orchestra.run.smithi186.stdout:3686 read 0xce400 thru 0xcee34 (0xa35 bytes) 2024-05-06T00:28:19.183 INFO:teuthology.orchestra.run.smithi186.stdout:3689 read 0x164c00 thru 0x171831 (0xcc32 bytes) 2024-05-06T00:28:19.197 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:19.197+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:19.204 INFO:teuthology.orchestra.run.smithi186.stdout:3690 trunc from 0x230600 to 0x8af9800 2024-05-06T00:28:19.221 INFO:teuthology.orchestra.run.smithi186.stdout:3691 write 0x85ce200 thru 0x85d2355 (0x4156 bytes) 2024-05-06T00:28:19.223 INFO:teuthology.orchestra.run.smithi186.stdout:3694 write 0x2849800 thru 0x2854637 (0xae38 bytes) 2024-05-06T00:28:19.223 INFO:teuthology.orchestra.run.smithi186.stdout:3695 read 0x324ea00 thru 0x32596af (0xacb0 bytes) 2024-05-06T00:28:19.223 INFO:teuthology.orchestra.run.smithi186.stdout:3698 punch from 0x2b52000 to 0x2b61000, (0xf000 bytes) 2024-05-06T00:28:19.224 INFO:teuthology.orchestra.run.smithi186.stdout:3699 read 0x2f33000 thru 0x2f33b21 (0xb22 bytes) 2024-05-06T00:28:19.229 INFO:teuthology.orchestra.run.smithi186.stdout:3702 trunc from 0x8af9800 to 0xb189400 2024-05-06T00:28:19.270 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.809204440999999, 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:28:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:28:19.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.808402452000003, 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.808315997999998, 2024-05-06T00:28:19.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.284 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.808243404999999, 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.285 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:28:19.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:28:19.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.808183554000003, 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:28:19.289 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:28:19.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:28:19.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.808107870999997, 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.298 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:28:19.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.298 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:19.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2967 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 159744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.800483882999998, 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026274390000000002, 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2967 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.302 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467897+0000", 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467898+0000", 2024-05-06T00:28:19.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.046e-06 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467917+0000", 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8896e-05 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467937+0000", 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9613999999999999e-05 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467961+0000", 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.419e-05 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.493935+0000", 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025973922 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494053+0000", 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011866099999999999 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494171+0000", 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011806099999999999 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.800402298000002, 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.306 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.310 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.800348091000004, 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:28:19.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.800295941999998, 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:28:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3040 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:19.319 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.798440620000001, 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026298018999999999, 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3040 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469940+0000", 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469941+0000", 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0979999999999999e-06 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469946+0000", 2024-05-06T00:28:19.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0170000000000004e-06 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469963+0000", 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6541999999999998e-05 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.469977+0000", 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4015999999999999e-05 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495954+0000", 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025976902999999999 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496101+0000", 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147529 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496238+0000", 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136914 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.798338227000002, 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:28:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.798261887000002, 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:28:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.983124179999997, 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:28:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.983071739000003, 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:28:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:28:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:28:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.976051755999997, 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:28:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.976005045000001, 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.975674937000001, 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:28:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.975647015, 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:28:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:28:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.964780126000001, 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:19.370 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:19.373 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:19.366+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:19.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3703 trunc from 0xb189400 to 0x856e00 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3704 read 0x696200 thru 0x69be53 (0x5c54 bytes) 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3706 trunc from 0x856e00 to 0x64a5600 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3707 punch from 0x16ba800 to 0x16bf400, (0x4c00 bytes) 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3708 punch from 0x51c4a00 to 0x51cde00, (0x9400 bytes) 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3709 read 0x1214400 thru 0x12189ec (0x45ed bytes) 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3711 read 0x591ea00 thru 0x5923c38 (0x5239 bytes) 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3713 write 0x6206200 thru 0x6207864 (0x1665 bytes) 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3716 read 0x362b000 thru 0x362f034 (0x4035 bytes) 2024-05-06T00:28:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:3717 write 0x8964e00 thru 0x896c972 (0x7b73 bytes) 2024-05-06T00:28:19.393 INFO:teuthology.orchestra.run.smithi186.stdout:3718 write 0xb443200 thru 0xb450846 (0xd647 bytes) 2024-05-06T00:28:19.395 INFO:teuthology.orchestra.run.smithi186.stdout:3719 punch from 0x8de7800 to 0x8de8e00, (0x1600 bytes) 2024-05-06T00:28:19.417 INFO:teuthology.orchestra.run.smithi186.stdout:3720 trunc from 0xb450a00 to 0xb4a00 2024-05-06T00:28:19.473 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:19.470+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:19.474 INFO:teuthology.orchestra.run.smithi186.stdout:3722 write 0x9c0c200 thru 0x9c121c5 (0x5fc6 bytes) 2024-05-06T00:28:19.477 INFO:teuthology.orchestra.run.smithi186.stdout:3723 read 0x4ef0400 thru 0x4efb143 (0xad44 bytes) 2024-05-06T00:28:19.477 INFO:teuthology.orchestra.run.smithi186.stdout:3726 trunc from 0x9c12200 to 0x9edf200 2024-05-06T00:28:19.501 INFO:teuthology.orchestra.run.smithi186.stdout:3727 read 0x17b2800 thru 0x17b371b (0xf1c bytes) 2024-05-06T00:28:19.506 INFO:teuthology.orchestra.run.smithi186.stdout:3729 read 0x4aa5e00 thru 0x4aade81 (0x8082 bytes) 2024-05-06T00:28:19.506 INFO:teuthology.orchestra.run.smithi186.stdout:3730 read 0x1074200 thru 0x107ba17 (0x7818 bytes) 2024-05-06T00:28:19.507 INFO:teuthology.orchestra.run.smithi186.stdout:3732 read 0x78c8a00 thru 0x78cdf7e (0x557f bytes) 2024-05-06T00:28:19.507 INFO:teuthology.orchestra.run.smithi186.stdout:3733 trunc from 0x9edf200 to 0x962b400 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5263.0:1864 3.1f 3.99ea875f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:19.519462+0000", 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.8978e-05, 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121278, 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:19.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5263", 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/3584077954", 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1864 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:19.519462+0000", 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:19.519462+0000", 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:19.519463+0000", 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.274e-06 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:19.519470+0000", 2024-05-06T00:28:19.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1930000000000004e-06 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:19.519484+0000", 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.365e-05 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:19.519495+0000", 2024-05-06T00:28:19.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0866999999999999e-05 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:19.519576+0000", 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1120999999999998e-05 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 1 2024-05-06T00:28:19.522 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:19.531 INFO:teuthology.orchestra.run.smithi186.stdout:3736 write 0x4446800 thru 0x444c936 (0x6137 bytes) 2024-05-06T00:28:19.533 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:19.548 INFO:teuthology.orchestra.run.smithi186.stdout:3737 trunc from 0x962b400 to 0xd989e00 2024-05-06T00:28:19.566 INFO:teuthology.orchestra.run.smithi186.stdout:3739 punch from 0x4231800 to 0x4233400, (0x1c00 bytes) 2024-05-06T00:28:19.576 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:19.577+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:19.577 INFO:teuthology.orchestra.run.smithi186.stdout:3741 trunc from 0xd989e00 to 0xa84ba00 2024-05-06T00:28:19.590 INFO:teuthology.orchestra.run.smithi186.stdout:3742 punch from 0x713f800 to 0x7148600, (0x8e00 bytes) 2024-05-06T00:28:19.609 INFO:teuthology.orchestra.run.smithi186.stdout:3743 write 0xcb6bc00 thru 0xcb79872 (0xdc73 bytes) 2024-05-06T00:28:19.610 INFO:teuthology.orchestra.run.smithi186.stdout:3745 write 0x4570a00 thru 0x457807a (0x767b bytes) 2024-05-06T00:28:19.624 INFO:teuthology.orchestra.run.smithi186.stdout:3746 trunc from 0xcb79a00 to 0x7756400 2024-05-06T00:28:19.656 INFO:teuthology.orchestra.run.smithi186.stdout:3748 read 0x256aa00 thru 0x256cfbc (0x25bd bytes) 2024-05-06T00:28:19.658 INFO:teuthology.orchestra.run.smithi186.stdout:3750 punch from 0x4a12600 to 0x4a12a00, (0x400 bytes) 2024-05-06T00:28:19.658 INFO:teuthology.orchestra.run.smithi186.stdout:3751 trunc from 0x7756400 to 0x755c000 2024-05-06T00:28:19.675 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:19.670+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:19.683 INFO:teuthology.orchestra.run.smithi186.stdout:3752 write 0x3655a00 thru 0x365f670 (0x9c71 bytes) 2024-05-06T00:28:19.685 INFO:teuthology.orchestra.run.smithi186.stdout:3757 write 0x263a000 thru 0x2642e5f (0x8e60 bytes) 2024-05-06T00:28:19.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:19.714 INFO:teuthology.orchestra.run.smithi186.stdout:3760 write 0x93a4400 thru 0x93a4b0e (0x70f bytes) 2024-05-06T00:28:19.715 INFO:teuthology.orchestra.run.smithi186.stdout:3763 read 0xeae200 thru 0xeb8299 (0xa09a bytes) 2024-05-06T00:28:19.737 INFO:teuthology.orchestra.run.smithi186.stdout:3766 trunc from 0x93a4c00 to 0x91ba00 2024-05-06T00:28:19.776 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:19.770+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:19.789 INFO:teuthology.orchestra.run.smithi186.stdout:3767 write 0x1a64400 thru 0x1a6a24e (0x5e4f bytes) 2024-05-06T00:28:19.817 INFO:teuthology.orchestra.run.smithi186.stdout:3768 write 0x3ec5000 thru 0x3ec74ab (0x24ac bytes) 2024-05-06T00:28:19.856 INFO:teuthology.orchestra.run.smithi186.stdout:3769 write 0xaab5200 thru 0xaabe382 (0x9183 bytes) 2024-05-06T00:28:19.858 INFO:teuthology.orchestra.run.smithi186.stdout:3771 read 0x79e9600 thru 0x79ec0cb (0x2acc bytes) 2024-05-06T00:28:19.877 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:19.877+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:19.882 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:19.883 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:19.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:19.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:19.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.884 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:19.884 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:19.885 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 81.075817009999994, 2024-05-06T00:28:19.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:28:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:19.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:28:19.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.887 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:19.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:19.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:28:19.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:28:19.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:28:19.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:28:19.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:19.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:28:19.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:28:19.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:28:19.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:28:19.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:28:19.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:28:19.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:28:19.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:28:19.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:28:19.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:28:19.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.899 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.900 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:19.900 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:19.900 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 81.075800806999993, 2024-05-06T00:28:19.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:28:19.901 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:28:19.902 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:28:19.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:28:19.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.902 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:19.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:19.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:28:19.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:28:19.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:28:19.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:28:19.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:28:19.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:28:19.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:28:19.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:28:19.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:28:19.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:28:19.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:28:19.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:28:19.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:28:19.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:28:19.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:19.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:28:19.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:28:19.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34113 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 845824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067670724999999, 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.073299384, 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34113 2024-05-06T00:28:19.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2200000000000004e-07 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812509+0000", 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-06 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812519+0000", 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6979999999999993e-06 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812528+0000", 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5370000000000003e-06 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817889+0000", 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053609809999999999 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817903+0000", 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3545e-05 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885659+0000", 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.067755797 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885734+0000", 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5698999999999995e-05 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885805+0000", 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0442000000000001e-05 2024-05-06T00:28:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34114 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 851968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067638689000001, 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0734813590000001, 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34114 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-07 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812540+0000", 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4509999999999999e-06 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5069999999999997e-06 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812552+0000", 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817936+0000", 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053840650000000004 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817944+0000", 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5890000000000003e-06 2024-05-06T00:28:19.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885879+0000", 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0679342119999999 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885952+0000", 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2924999999999997e-05 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886019+0000", 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7358000000000001e-05 2024-05-06T00:28:19.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34115 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 858112~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.06760358, 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0736428419999999, 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34115 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9700000000000002e-07 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812575+0000", 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.402e-06 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812587+0000", 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7979999999999996e-06 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817971+0000", 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053842719999999998 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817979+0000", 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4250000000000001e-06 2024-05-06T00:28:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886063+0000", 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068084475 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886129+0000", 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6109e-05 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886216+0000", 2024-05-06T00:28:19.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6312999999999998e-05 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067570590000001, 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:19.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:28:19.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:19.939 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.06753922, 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:28:19.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:19.944 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067505431000001, 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:28:19.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067324975, 2024-05-06T00:28:19.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:19.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.954 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067291332, 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:28:19.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:28:19.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:28:19.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:28:19.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:28:19.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:28:19.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.961 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.962 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.962 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.06726755, 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:28:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.967 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067241273, 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:28:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067198032, 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:28:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067168403, 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:28:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:28:19.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:28:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:28:19.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067135553, 2024-05-06T00:28:19.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:19.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:28:19.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:28:19.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.988 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067103915000001, 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:19.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:28:19.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067071113000001, 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:19.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:19.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:19.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:28:19.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:19.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:28:19.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:28:19.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:19.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:28:19.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:28:19.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:19.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:19.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:20.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:28:20.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:28:20.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:20.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:28:20.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:28:20.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067039080000001, 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:28:20.001 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:28:20.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:28:20.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:20.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:20.006 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.067007001, 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:20.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:20.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:20.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.066975693, 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:20.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:28:20.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:28:20.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.016 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:20.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:20.016 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:20.016 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:20.016 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 0] out_osds: [5, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:20.016 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:20.016 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-05-06T00:28:20.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-05-06T00:28:20.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:20.020 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:28:20.021 INFO:teuthology.orchestra.run.smithi186.stdout:3773 trunc from 0xaabe400 to 0x1cd8000 2024-05-06T00:28:20.021 INFO:teuthology.orchestra.run.smithi186.stdout:3774 read 0xde0400 thru 0xdef79f (0xf3a0 bytes) 2024-05-06T00:28:20.021 INFO:teuthology.orchestra.run.smithi186.stdout:3775 trunc from 0x1cd8000 to 0xb4ecc00 2024-05-06T00:28:20.021 INFO:teuthology.orchestra.run.smithi186.stdout:3776 trunc from 0xb4ecc00 to 0xb83bc00 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3777 write 0x3f0c800 thru 0x3f1a900 (0xe101 bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3779 read 0x7d02e00 thru 0x7d0ef85 (0xc186 bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3780 read 0x76c5400 thru 0x76c722e (0x1e2f bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3781 read 0xa6df000 thru 0xa6df00a (0xb bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3784 trunc from 0xb83bc00 to 0xcf11a00 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3785 trunc from 0xcf11a00 to 0xdc38000 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3787 write 0x6fe3e00 thru 0x6fed01f (0x9220 bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3789 read 0x1bfd800 thru 0x1c081d9 (0xa9da bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3791 write 0xc699c00 thru 0xc69c070 (0x2471 bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3792 write 0x3847800 thru 0x384841b (0xc1c bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3794 read 0x34d8a00 thru 0x34d9caa (0x12ab bytes) 2024-05-06T00:28:20.022 INFO:teuthology.orchestra.run.smithi186.stdout:3795 write 0x1c84e00 thru 0x1c91823 (0xca24 bytes) 2024-05-06T00:28:20.023 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:20.014+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:20.047 INFO:teuthology.orchestra.run.smithi186.stdout:3796 write 0xe895e00 thru 0xe896159 (0x35a bytes) 2024-05-06T00:28:20.047 INFO:teuthology.orchestra.run.smithi186.stdout:3797 write 0x2aa1a00 thru 0x2aafd49 (0xe34a bytes) 2024-05-06T00:28:20.057 INFO:teuthology.orchestra.run.smithi186.stdout:3800 trunc from 0xe896200 to 0x9ae6600 2024-05-06T00:28:20.088 INFO:teuthology.orchestra.run.smithi186.stdout:3802 punch from 0x6d55e00 to 0x6d57e00, (0x2000 bytes) 2024-05-06T00:28:20.091 INFO:teuthology.orchestra.run.smithi186.stdout:3804 punch from 0x5c75200 to 0x5c7ce00, (0x7c00 bytes) 2024-05-06T00:28:20.091 INFO:teuthology.orchestra.run.smithi186.stdout:3806 write 0x224a400 thru 0x224c601 (0x2202 bytes) 2024-05-06T00:28:20.091 INFO:teuthology.orchestra.run.smithi186.stdout:3807 trunc from 0x9ae6600 to 0x9a3cc00 2024-05-06T00:28:20.112 INFO:teuthology.orchestra.run.smithi186.stdout:3808 read 0x5c8a600 thru 0x5c8ff2e (0x592f bytes) 2024-05-06T00:28:20.114 INFO:teuthology.orchestra.run.smithi186.stdout:3810 write 0x45da600 thru 0x45e5ab5 (0xb4b6 bytes) 2024-05-06T00:28:20.118 INFO:teuthology.orchestra.run.smithi186.stdout:3811 read 0x474c600 thru 0x475913c (0xcb3d bytes) 2024-05-06T00:28:20.119 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:20.114+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:20.127 INFO:teuthology.orchestra.run.smithi186.stdout:3812 trunc from 0x9a3cc00 to 0xda0ee00 2024-05-06T00:28:20.159 INFO:teuthology.orchestra.run.smithi186.stdout:3813 write 0xe0dd800 thru 0xe0e9624 (0xbe25 bytes) 2024-05-06T00:28:20.160 INFO:teuthology.orchestra.run.smithi186.stdout:3817 write 0x63e3a00 thru 0x63ec422 (0x8a23 bytes) 2024-05-06T00:28:20.160 INFO:teuthology.orchestra.run.smithi186.stdout:3818 punch from 0x5d2b000 to 0x5d32000, (0x7000 bytes) 2024-05-06T00:28:20.163 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:20.183 INFO:teuthology.orchestra.run.smithi186.stdout:3820 trunc from 0xe0e9800 to 0x19d7c00 2024-05-06T00:28:20.204 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:20.204 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:20.219 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:20.214+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:20.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:20.249 INFO:teuthology.orchestra.run.smithi186.stdout:3822 write 0xcb2fa00 thru 0xcb314a0 (0x1aa1 bytes) 2024-05-06T00:28:20.250 INFO:teuthology.orchestra.run.smithi186.stdout:3823 read 0x77b6200 thru 0x77bc87e (0x667f bytes) 2024-05-06T00:28:20.251 INFO:teuthology.orchestra.run.smithi186.stdout:3827 read 0xb017a00 thru 0xb020d8d (0x938e bytes) 2024-05-06T00:28:20.275 INFO:teuthology.orchestra.run.smithi186.stdout:3828 write 0xe11d200 thru 0xe128f10 (0xbd11 bytes) 2024-05-06T00:28:20.277 INFO:teuthology.orchestra.run.smithi186.stdout:3829 write 0x5ff4400 thru 0x6001205 (0xce06 bytes) 2024-05-06T00:28:20.277 INFO:teuthology.orchestra.run.smithi186.stdout:3830 write 0x85ca800 thru 0x85d64f3 (0xbcf4 bytes) 2024-05-06T00:28:20.278 INFO:teuthology.orchestra.run.smithi186.stdout:3835 write 0x8539200 thru 0x8547296 (0xe097 bytes) 2024-05-06T00:28:20.278 INFO:teuthology.orchestra.run.smithi186.stdout:3836 punch from 0x6663000 to 0x6667200, (0x4200 bytes) 2024-05-06T00:28:20.306 INFO:teuthology.orchestra.run.smithi186.stdout:3839 write 0xecaca00 thru 0xecb101e (0x461f bytes) 2024-05-06T00:28:20.306 INFO:teuthology.orchestra.run.smithi186.stdout:3840 read 0x58c0e00 thru 0x58cb277 (0xa478 bytes) 2024-05-06T00:28:20.306 INFO:teuthology.orchestra.run.smithi186.stdout:3841 read 0x735a00 thru 0x740a93 (0xb094 bytes) 2024-05-06T00:28:20.307 INFO:teuthology.orchestra.run.smithi186.stdout:3847 write 0x84b0200 thru 0x84bf31d (0xf11e bytes) 2024-05-06T00:28:20.320 INFO:teuthology.orchestra.run.smithi186.stdout:3848 trunc from 0xecb1200 to 0x771cc00 2024-05-06T00:28:20.320 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:20.321+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:20.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:20.333 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:20.350 INFO:teuthology.orchestra.run.smithi186.stdout:3849 punch from 0x71e4a00 to 0x71e5400, (0xa00 bytes) 2024-05-06T00:28:20.352 INFO:teuthology.orchestra.run.smithi186.stdout:3852 read 0x492fc00 thru 0x493f1f3 (0xf5f4 bytes) 2024-05-06T00:28:20.352 INFO:teuthology.orchestra.run.smithi186.stdout:3857 punch from 0x2fc1c00 to 0x2fcec00, (0xd000 bytes) 2024-05-06T00:28:20.352 INFO:teuthology.orchestra.run.smithi186.stdout:3858 punch from 0x6571000 to 0x6572400, (0x1400 bytes) 2024-05-06T00:28:20.353 INFO:teuthology.orchestra.run.smithi186.stdout:3859 read 0x3ace400 thru 0x3ad5718 (0x7319 bytes) 2024-05-06T00:28:20.353 INFO:teuthology.orchestra.run.smithi186.stdout:3860 read 0x504600 thru 0x50c3f2 (0x7df3 bytes) 2024-05-06T00:28:20.358 INFO:teuthology.orchestra.run.smithi186.stdout:3862 trunc from 0x771cc00 to 0x529d800 2024-05-06T00:28:20.367 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:20.368 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:20.380 INFO:teuthology.orchestra.run.smithi186.stdout:3863 punch from 0x28fc000 to 0x2905e00, (0x9e00 bytes) 2024-05-06T00:28:20.382 INFO:teuthology.orchestra.run.smithi186.stdout:3869 read 0x128ca00 thru 0x1290fad (0x45ae bytes) 2024-05-06T00:28:20.382 INFO:teuthology.orchestra.run.smithi186.stdout:3871 write 0xe10600 thru 0xe15f38 (0x5939 bytes) 2024-05-06T00:28:20.395 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:20.395 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:20.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:20.399 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:20.408 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:20.422 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:20.418+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:20.423 INFO:teuthology.orchestra.run.smithi186.stdout:3874 write 0xd589400 thru 0xd59268d (0x928e bytes) 2024-05-06T00:28:20.423 INFO:teuthology.orchestra.run.smithi186.stdout:3875 read 0x8f49000 thru 0x8f5682f (0xd830 bytes) 2024-05-06T00:28:20.424 INFO:teuthology.orchestra.run.smithi186.stdout:3881 read 0x4ff3c00 thru 0x5000982 (0xcd83 bytes) 2024-05-06T00:28:20.424 INFO:teuthology.orchestra.run.smithi186.stdout:3885 punch from 0x6ff6800 to 0x7003600, (0xce00 bytes) 2024-05-06T00:28:20.425 INFO:teuthology.orchestra.run.smithi186.stdout:3886 write 0x54e1800 thru 0x54e7427 (0x5c28 bytes) 2024-05-06T00:28:20.425 INFO:teuthology.orchestra.run.smithi186.stdout:3889 write 0x397800 thru 0x3a2e31 (0xb632 bytes) 2024-05-06T00:28:20.425 INFO:teuthology.orchestra.run.smithi186.stdout:3890 read 0x972e600 thru 0x973dce6 (0xf6e7 bytes) 2024-05-06T00:28:20.426 INFO:teuthology.orchestra.run.smithi186.stdout:3891 punch from 0x29b7800 to 0x29c0c00, (0x9400 bytes) 2024-05-06T00:28:20.426 INFO:teuthology.orchestra.run.smithi186.stdout:3892 write 0xac35e00 thru 0xac3d8c6 (0x7ac7 bytes) 2024-05-06T00:28:20.427 INFO:teuthology.orchestra.run.smithi186.stdout:3893 write 0x79ba400 thru 0x79c9bce (0xf7cf bytes) 2024-05-06T00:28:20.427 INFO:teuthology.orchestra.run.smithi186.stdout:3896 write 0x24800 thru 0x2ef25 (0xa726 bytes) 2024-05-06T00:28:20.429 INFO:teuthology.orchestra.run.smithi186.stdout:3897 read 0x8462c00 thru 0x8463e36 (0x1237 bytes) 2024-05-06T00:28:20.429 INFO:teuthology.orchestra.run.smithi186.stdout:3899 read 0x3005200 thru 0x3008ce8 (0x3ae9 bytes) 2024-05-06T00:28:20.429 INFO:teuthology.orchestra.run.smithi186.stdout:3905 read 0x238fc00 thru 0x239e4b1 (0xe8b2 bytes) 2024-05-06T00:28:20.429 INFO:teuthology.orchestra.run.smithi186.stdout:3906 write 0x3401e00 thru 0x340871f (0x6920 bytes) 2024-05-06T00:28:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:20.457 INFO:teuthology.orchestra.run.smithi186.stdout:3907 write 0xe0c2800 thru 0xe0c994b (0x714c bytes) 2024-05-06T00:28:20.458 INFO:teuthology.orchestra.run.smithi186.stdout:3908 write 0x5e6ec00 thru 0x5e7839b (0x979c bytes) 2024-05-06T00:28:20.459 INFO:teuthology.orchestra.run.smithi186.stdout:3910 write 0xd9d8000 thru 0xd9e1e31 (0x9e32 bytes) 2024-05-06T00:28:20.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:20.471 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:20.471 INFO:teuthology.orchestra.run.smithi186.stdout:3911 trunc from 0xe0c9a00 to 0x80ec600 2024-05-06T00:28:20.503 INFO:teuthology.orchestra.run.smithi186.stdout:3913 punch from 0x6323200 to 0x632c800, (0x9600 bytes) 2024-05-06T00:28:20.505 INFO:teuthology.orchestra.run.smithi186.stdout:3915 punch from 0x7587000 to 0x7594c00, (0xdc00 bytes) 2024-05-06T00:28:20.506 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-06T00:28:20.505+0000 7fdfb36a9640 -1 mon.a@0(leader).osd e95 definitely_dead 0 2024-05-06T00:28:20.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:20.506 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:20.526 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:20.518+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:20.531 INFO:teuthology.orchestra.run.smithi186.stdout:3920 write 0xd34d200 thru 0xd34d67b (0x47c bytes) 2024-05-06T00:28:20.533 INFO:teuthology.orchestra.run.smithi186.stdout:3921 write 0x2147e00 thru 0x2155307 (0xd508 bytes) 2024-05-06T00:28:20.534 INFO:teuthology.orchestra.run.smithi186.stdout:3922 read 0x9fbae00 thru 0x9fca68f (0xf890 bytes) 2024-05-06T00:28:20.534 INFO:teuthology.orchestra.run.smithi186.stdout:3923 punch from 0x680de00 to 0x6813600, (0x5800 bytes) 2024-05-06T00:28:20.535 INFO:teuthology.orchestra.run.smithi186.stdout:3925 read 0x5028c00 thru 0x5031ce7 (0x90e8 bytes) 2024-05-06T00:28:20.536 INFO:teuthology.orchestra.run.smithi186.stdout:3931 read 0xca93600 thru 0xca9a3a2 (0x6da3 bytes) 2024-05-06T00:28:20.536 INFO:teuthology.orchestra.run.smithi186.stdout:3932 read 0x2bf2600 thru 0x2bf8a98 (0x6499 bytes) 2024-05-06T00:28:20.536 INFO:teuthology.orchestra.run.smithi186.stdout:3934 punch from 0x6036a00 to 0x603c000, (0x5600 bytes) 2024-05-06T00:28:20.537 INFO:teuthology.orchestra.run.smithi186.stdout:3935 write 0x4a4b800 thru 0x4a5aff6 (0xf7f7 bytes) 2024-05-06T00:28:20.537 INFO:teuthology.orchestra.run.smithi186.stdout:3937 clone 25 order 20 su 65536 sc 11 2024-05-06T00:28:20.561 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:20.561 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:20.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:20.660 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:20.661+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:20.724 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:20.718+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:20.730 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:20.730 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:20.745 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:20.825 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:20.822+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:20.910 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:20.911 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:20.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:20.926 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:20.922+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:20.954 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.5. 2024-05-06T00:28:20.981 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-05-06T00:28:21.027 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:21.022+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:21.051 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:21.051 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:21.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:21.128 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:21.122+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:21.198 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:21.199 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:21.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:21.257 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:21.254+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:21.330 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:21.329+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:21.379 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:21.379 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:21.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:21.431 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:21.433+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:21.510 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:21.510 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:21.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:21.569 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:21.566+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:21.633 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:21.633+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:21.651 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:21.652 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:21.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:21.734 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:21.733+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:21.832 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:21.833 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:21.835 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:21.830+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:21.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:21.936 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:21.933+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:22.014 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:22.015 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:22.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:28:22.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:22.073 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:22.070+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:22.139 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:22.134+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:22.151 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:22.151 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:22.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:22.239 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:22.237+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:22.326 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:22.327 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:22.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:22.341 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:22.334+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:22.441 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:22.438+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:22.507 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:22.507 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:22.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:22.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:22.568 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:22.569+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:22.608 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:28:22.643 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:22.641+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:22.664 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:22.664 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:22.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:22.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:22.720 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:22.737 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:22.744 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:22.741+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:22.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:22.775 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:22.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:22.795 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:22.816 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:22.835 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:22.835 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:22.835 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:22.835 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:22.835 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:22.845 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:22.842+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:22.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:22.946 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:22.945+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.221863425999999, 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:28:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:28:23.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:23.033 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.221847222999997, 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:28:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34113 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 845824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:23.039 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213717141, 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.073299384, 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34113 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:23.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2200000000000004e-07 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812509+0000", 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-06 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812519+0000", 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6979999999999993e-06 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812528+0000", 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5370000000000003e-06 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817889+0000", 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053609809999999999 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817903+0000", 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3545e-05 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885659+0000", 2024-05-06T00:28:23.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.067755797 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885734+0000", 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5698999999999995e-05 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885805+0000", 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0442000000000001e-05 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34114 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 851968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213685105, 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0734813590000001, 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34114 2024-05-06T00:28:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-07 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812540+0000", 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4509999999999999e-06 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5069999999999997e-06 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812552+0000", 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817936+0000", 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053840650000000004 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817944+0000", 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5890000000000003e-06 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885879+0000", 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0679342119999999 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885952+0000", 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2924999999999997e-05 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886019+0000", 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7358000000000001e-05 2024-05-06T00:28:23.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34115 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 858112~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213649996000001, 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0736428419999999, 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34115 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9700000000000002e-07 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812575+0000", 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.402e-06 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812587+0000", 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7979999999999996e-06 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817971+0000", 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053842719999999998 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817979+0000", 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4250000000000001e-06 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886063+0000", 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068084475 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886129+0000", 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6109e-05 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886216+0000", 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6312999999999998e-05 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213617006, 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:28:23.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:28:23.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213585636000001, 2024-05-06T00:28:23.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:28:23.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:28:23.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:23.065 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213551847000002, 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:23.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:28:23.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213371390999999, 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213337748000001, 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:28:23.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213313966000001, 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:28:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213287689000001, 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213244448000001, 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:28:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:28:23.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213214819000001, 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:28:23.096 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:28:23.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:28:23.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:23.101 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213181969000001, 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:23.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.106 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213150331000001, 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:28:23.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213117529000002, 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:28:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213085496000001, 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:28:23.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213053417000001, 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:28:23.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:28:23.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.213022109000001, 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.127 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:28:23.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.131 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:23.132 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone24 from 0xd34d800 (overlap 0xb4a00) to 0x48400 2024-05-06T00:28:23.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:23.134 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:23.130+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:23.233 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:23.233+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:23.310 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5276.0:1 3.e 3:701e3dcd:::rbd_id.image_client.0-clone25:head [stat out=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98)", 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:23.312154+0000", 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.00019092800000000001, 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026121400000000001, 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5276", 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2173543964", 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1 2024-05-06T00:28:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312154+0000", 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312154+0000", 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312156+0000", 2024-05-06T00:28:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1469999999999999e-06 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312166+0000", 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.893e-06 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312188+0000", 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1593000000000001e-05 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312203+0000", 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5302999999999999e-05 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312271+0000", 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7951e-05 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312332+0000", 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0798000000000003e-05 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:23.312452+0000", 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000119626 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 1 2024-05-06T00:28:23.315 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:23.326 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:23.372 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:23.373+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:23.435 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:23.430+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:23.503 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:23.503 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:23.503 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:23.504 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:23.504 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:23.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:23.528 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-05-06T00:28:23.536 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:23.530+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:23.637 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:23.637+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21566 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 417792~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622528178, 2024-05-06T00:28:23.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090303018999999998, 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21566 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5400000000000002e-07 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026960+0000", 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4449999999999999e-06 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026966+0000", 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6200000000000001e-06 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026978+0000", 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1248e-05 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116885+0000", 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.089907163999999998 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116986+0000", 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101363 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117255+0000", 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026882499999999999 2024-05-06T00:28:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21568 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 423936~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622420692999999, 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090403081999999996, 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21568 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027060+0000", 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027063+0000", 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3249999999999999e-06 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027072+0000", 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2859999999999999e-06 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027078+0000", 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117286+0000", 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090207976999999995 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117372+0000", 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5865000000000005e-05 2024-05-06T00:28:23.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117463+0000", 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0662999999999996e-05 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21570 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 430080~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622394084, 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090545366000000002, 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21570 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027087+0000", 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3799999999999998e-07 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027089+0000", 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.12e-06 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027095+0000", 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4590000000000003e-06 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027100+0000", 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117492+0000", 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090391632999999999 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117546+0000", 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4177e-05 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117631+0000", 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5173999999999995e-05 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.664 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21572 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 436224~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622371723000001, 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090693918999999998, 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21572 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027109+0000", 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7100000000000002e-07 2024-05-06T00:28:23.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027112+0000", 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5490000000000001e-06 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027119+0000", 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7829999999999999e-06 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027124+0000", 2024-05-06T00:28:23.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.04e-06 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117662+0000", 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090537113000000002 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117717+0000", 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5798999999999999e-05 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117802+0000", 2024-05-06T00:28:23.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5063999999999997e-05 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622347656999999, 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.669 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:23.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:28:23.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622326868000002, 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:28:23.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.677 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622306806000001, 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:28:23.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:28:23.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:28:23.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622287530000001, 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.682 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:23.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:23.686 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622267411999999, 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:23.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:28:23.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622236003000001, 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.691 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:28:23.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622203090999999, 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.699 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622182504000001, 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.703 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622163124, 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:28:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:28:23.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622118443000002, 2024-05-06T00:28:23.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:28:23.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622053352000002, 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:23.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.622021778000001, 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.721 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.621999516999999, 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:28:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.621978401, 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:28:23.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.964261775000001, 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:28:23.730 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:28:23.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.963851533, 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:28:23.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:23.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:28:23.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:23.739 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:23.741 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:23.741 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:23.738+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:23.842 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:23.841+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:23.937 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:23.937 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:23.937 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:23.937 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:23.943 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:23.938+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:23.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:24.044 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:24.038+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:24.117 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:24.117 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:24.117 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:24.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:24.176 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:24.177+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:24.246 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:24.245+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:24.263 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:24.263 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.257123855000003, 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:24.264 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:28:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.256895282999999, 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:28:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:28:24.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:28:24.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.256817998999999, 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:28:24.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.256622018000002, 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:24.277 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:28:24.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:28:24.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.349512642000001, 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:28:24.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:28:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.349451763000001, 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:24.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:28:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.231895312999999, 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.290 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:28:24.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.231850585, 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:28:24.294 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:28:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.231812183999999, 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:28:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.231774922, 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.231738013000001, 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:28:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:28:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.231700095000001, 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.231662979999999, 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.225882531, 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.225844447, 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.325 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:24.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.225806772999999, 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:24.329 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:24.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:28:24.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:24.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:24.333 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.216371262999999, 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:24.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:24.337 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.216338835999998, 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:24.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:24.341 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.216314954000001, 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:24.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:28:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:28:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.216295024000001, 2024-05-06T00:28:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:24.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:28:24.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:24.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:24.349 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:24.351 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:24.351 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:24.346+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:24.451 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:24.446+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:24.488 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:24.488 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:24.488 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:24.488 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:24.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:24.593 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:24.590+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:24.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:28:24.670 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:24.671 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:24.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:24.692 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:24.693+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:24.773 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:24.770+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:24.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:24.855 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:24.850+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:24.860 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:24.860 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:24.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:24.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 115.255300152, 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:28:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:28:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:28:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.982651453000003, 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:28:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.982616277000005, 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:28:24.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.982527598000004, 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:28:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.982425655, 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:24.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:24.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:28:24.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:28:24.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.982316553000004, 2024-05-06T00:28:24.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:24.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:28:24.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:28:24.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.898 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.982218693999997, 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:28:24.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:28:24.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.982153643999993, 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.905 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:28:24.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:28:24.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.910 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.982060997999994, 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:28:24.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:28:24.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.916 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.981988994000005, 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.917 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:28:24.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:28:24.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.922 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222527507999999, 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:28:24.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.928 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222508259999998, 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:24.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222458140000001, 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.934 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222437088999996, 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.940 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:24.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:28:24.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:28:24.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222421769999997, 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.951 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222404888, 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:24.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:28:24.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222386552000003, 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222370549999994, 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:24.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:28:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.222284187, 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:28:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.212321711000001, 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:24.980 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:24.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:24.982 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:24.978+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:24.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:25.012 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:25.012 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:25.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:25.030 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:25.051 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:25.051 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:25.072 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:25.082 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:25.078+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:25.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:25.158 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:25.158 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:25.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:25.183 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:25.181+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:25.285 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:25.285+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:25.340 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:25.340 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:25.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:25.386 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:25.385+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:25.477 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:25.477 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:25.487 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:25.485+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:25.491 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:25.588 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:25.582+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:25.608 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:25.608 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:25.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:25.689 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:25.689+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:25.785 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:25.786 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:25.790 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:25.786+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:25.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:25.891 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:25.889+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:25.964 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:28:25.966 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:25.966 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:25.981 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 0, 5] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:25.982 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:25.982 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-05-06T00:28:25.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-05-06T00:28:25.986 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:25.998 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:25.997+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:26.092 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:26.090+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:26.167 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:26.167 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:26.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:26.193 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:26.190+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:26.294 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:26.293+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:26.296 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:26.296 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:26.301 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-06T00:28:26.301+0000 7fdfb36a9640 -1 mon.a@0(leader).osd e101 definitely_dead 0 2024-05-06T00:28:26.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:26.395 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:26.394+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:26.472 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:26.473 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:26.488 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:26.496 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:26.494+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:26.597 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:26.594+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:26.610 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:26.610 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:26.624 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:26.698 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:26.698+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:26.744 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:26.744 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:26.759 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:26.805 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:26.805+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:26.880 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:26.881 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:26.894 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:26.937 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:26.937+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:26.973 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.3. 2024-05-06T00:28:26.999 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:26.997+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:27.000 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-05-06T00:28:27.065 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:27.065 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:27.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:27.101 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:27.098+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:27.111 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:27.200 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:27.200 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:27.201 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:27.201+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:27.215 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:27.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:27.274 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:27.291 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:27.291 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:27.302 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:27.298+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:27.308 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:27.329 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:27.329 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:27.350 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:27.350 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:27.371 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:27.371 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:27.372 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:27.372 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:27.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:27.372 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:27.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.372 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:27.372 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 59.366283817999999, 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:28:27.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 59.366055246000002, 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 59.365977962000002, 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 59.365781980999998, 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.458672605, 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:28:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:28:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.458611726000001, 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:28:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.341055275999999, 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:28:27.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.341010548, 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:28:27.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:28:27.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.340972146999999, 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.340934884999999, 2024-05-06T00:28:27.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:28:27.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:28:27.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.340897976000001, 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.417 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:28:27.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:28:27.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:28:27.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.340860058000001, 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.421 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:28:27.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:28:27.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.340822942999999, 2024-05-06T00:28:27.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.429 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.335042494, 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:28:27.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.33500441, 2024-05-06T00:28:27.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:28:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.334966735999998, 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:28:27.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:28:27.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.325531225999999, 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.443 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:27.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.325498799000002, 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:28:27.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:28:27.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.325474917000001, 2024-05-06T00:28:27.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:28:27.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.325454987000001, 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:28:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:27.459 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:27.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:27.462 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:27.458+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:27.562 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:27.558+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:27.592 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:27.593 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:27.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:27.693 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:27.690+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:27.764 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:27.758+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:27.768 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:27.769 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:27.769 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:27.769 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:27.769 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:27.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:27.873 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:27.870+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:27.951 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 118.34843266599999, 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:27.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:27.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:28:27.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:27.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:27.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075783967000007, 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:28:27.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:27.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.964 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075748790999995, 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:28:27.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075660111999994, 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:27.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:28:27.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:28:27.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:28:27.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:28:27.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075558169000004, 2024-05-06T00:28:27.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:28:27.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:28:27.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:27.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:27.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075449066999994, 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:27.982 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:27.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:28:27.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:27.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075351208000001, 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:27.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:28:27.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075286157999997, 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:27.994 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:28:27.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:28:27.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:27.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:28.000 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075193511999998, 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:28:28.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:28:28.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.075121507999995, 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.007 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:28:28.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:28:28.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315660022000003, 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:28:28.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:28:28.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315640774000002, 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:28:28.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:28:28.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315590654000005, 2024-05-06T00:28:28.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:28:28.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.031 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315569603, 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:28:28.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:28:28.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315554284000001, 2024-05-06T00:28:28.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:28.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:28:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315537402000004, 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:28:28.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:28:28.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315519065999993, 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.049 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:28:28.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315503063999998, 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:28.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:28:28.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.315416701000004, 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:28:28.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:28:28.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:28:28.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.066 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.305454225000005, 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:28:28.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:28:28.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.072 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:28.073 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:28:28.074 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:28.073+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:28.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:28.175 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:28.173+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:28.246 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:28.246 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:28.246 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:28.246 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:28.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:28.276 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:28.277+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:28.377 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:28.374+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:28.379 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:28.379 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:28.380 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:28.380 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:28.380 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:28.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:28.481 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:28.478+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:28.559 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:28:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.898608758000002, 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.919367253000001, 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:28:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:28:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.919293033999999, 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:28:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:28:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.919231135, 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.919171536, 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.919087400999999, 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:28:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.919021888, 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.586 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:28:28.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:28:28.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918964026000001, 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:28:28.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918867430999999, 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.595 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.599 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918789705999998, 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918716920000001, 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:28:28.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:28:28.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918639428999999, 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:28:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918545300000002, 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:28:28.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918462338000001, 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.617 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.621 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918388233000002, 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:28:28.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:28:28.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918289044000002, 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:28.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.630 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918225891999999, 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:28:28.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918125626999998, 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:28:28.635 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:28:28.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.918036255000001, 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.640 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:28:28.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.672177571999999, 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:28.644 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:28.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:28.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:28.648 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:28.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:28.650 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:28.649+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:28.751 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:28.749+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:28.819 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:28.820 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:28.820 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:28.820 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:28.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:28.852 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:28.849+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:28.950 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:28.951 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:28.952 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:28.946+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:28.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:28.978 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:28:29.052 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:29.053+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:29.131 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:29.131 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:29.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:29.131 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:29.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.131 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.132 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:29.132 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.669934472999998, 2024-05-06T00:28:29.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:28:29.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:28:29.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:28:29.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:28:29.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:28:29.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:28:29.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:28:29.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.669132484000002, 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:28:29.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.669046030000004, 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:28:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.668973437000005, 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.148 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:28:29.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.668913586000002, 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:28:29.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:28:29.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.668837902999996, 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.157 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:28:29.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:28:29.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.661132330000001, 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:28:29.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:28:29.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.661078122999996, 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.167 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.661025973999998, 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.171 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:28:29.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.659068258999994, 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:28:29.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:28:29.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.658991919000002, 2024-05-06T00:28:29.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:28:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:28:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:28:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:28:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.843854211999997, 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:28:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:28:29.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:28:29.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.196 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.843801771000003, 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.197 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:28:29.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:29.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.836781787999996, 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:28:29.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:28:29.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.836735077, 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:28:29.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:28:29.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.836404969, 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:28:29.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:28:29.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:28:29.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:28:29.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:28:29.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:28:29.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:28:29.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.836377046999999, 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:29.224 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:29.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:29.228 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.825510158, 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.232 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.1678603570000003, 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:28:29.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:28:29.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:28:29.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.1666757419999998, 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:29.238 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:28:29.242 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.243 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.243 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.243 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:29.243 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:29.243 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:29.244 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:29.238+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:29.245 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:29.345 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:29.342+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:29.390 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:29.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:29.391 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:29.391 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:29.406 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:29.445 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:29.442+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:29.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:29.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:29.537 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:29.537 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:29.537 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:29.537 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:29.548 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:29.542+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:29.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:29.602 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:29.602 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:29.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:29.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:29.637 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:29.648 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:29.649+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:29.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:29.658 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:29.678 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:29.678 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:29.680 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:29.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:29.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:29.750 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:29.746+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:29.827 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:29.827 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:29.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:29.888 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:29.889+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:29.913 INFO:teuthology.orchestra.run.smithi186.stdout:3939 trunc from 0xd34d800 to 0x1692800 2024-05-06T00:28:29.952 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:29.946+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:29.971 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:29.971 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:29.987 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:29.988 INFO:teuthology.orchestra.run.smithi186.stdout:3940 trunc from 0x1692800 to 0x27acc00 2024-05-06T00:28:30.026 INFO:teuthology.orchestra.run.smithi186.stdout:3941 trunc from 0x27acc00 to 0xb4fd000 2024-05-06T00:28:30.043 INFO:teuthology.orchestra.run.smithi186.stdout:3944 trunc from 0xb4fd000 to 0xa9a0400 2024-05-06T00:28:30.053 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:30.050+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:30.074 INFO:teuthology.orchestra.run.smithi186.stdout:3945 trunc from 0xa9a0400 to 0x58c2000 2024-05-06T00:28:30.093 INFO:teuthology.orchestra.run.smithi186.stdout:3946 read 0x2fc3e00 thru 0x2fcea41 (0xac42 bytes) 2024-05-06T00:28:30.096 INFO:teuthology.orchestra.run.smithi186.stdout:3948 write 0x1639e00 thru 0x16441ae (0xa3af bytes) 2024-05-06T00:28:30.154 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:30.150+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:30.155 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:30.155 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:30.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:30.255 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:30.253+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:30.291 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:30.292 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:30.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:30.398 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:30.397+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:30.460 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:30.457+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:30.475 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:30.475 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:30.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:30.577 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:30.577+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:30.662 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:30.658+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:30.665 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:30.665 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:30.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:30.763 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:30.761+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:30.822 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:30.822 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:30.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:30.881 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:30.878+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:30.964 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:30.964 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:30.965 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:30.965+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:30.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:31.011 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:28:31.034 INFO:teuthology.orchestra.run.smithi186.stdout:3949 write 0x1f54200 thru 0x1f615be (0xd3bf bytes) 2024-05-06T00:28:31.066 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:31.065+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:31.149 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:31.150 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:31.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:31.167 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:31.162+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:31.268 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:31.265+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:31.289 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:31.289 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:31.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:31.368 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:31.362+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:31.418 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:31.418 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:31.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:31.469 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:31.469+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:31.570 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:31.566+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:31.597 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:31.598 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:31.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:31.621 INFO:teuthology.orchestra.run.smithi186.stdout:3950 read 0x6ba00 thru 0x7a578 (0xeb79 bytes) 2024-05-06T00:28:31.622 INFO:teuthology.orchestra.run.smithi186.stdout:3952 read 0x5060a00 thru 0x506a0bc (0x96bd bytes) 2024-05-06T00:28:31.622 INFO:teuthology.orchestra.run.smithi186.stdout:3953 read 0x360b200 thru 0x36114e7 (0x62e8 bytes) 2024-05-06T00:28:31.622 INFO:teuthology.orchestra.run.smithi186.stdout:3956 read 0x18b0200 thru 0x18be753 (0xe554 bytes) 2024-05-06T00:28:31.622 INFO:teuthology.orchestra.run.smithi186.stdout:3962 punch from 0x4756000 to 0x475f000, (0x9000 bytes) 2024-05-06T00:28:31.623 INFO:teuthology.orchestra.run.smithi186.stdout:3964 read 0x33dfa00 thru 0x33eafe1 (0xb5e2 bytes) 2024-05-06T00:28:31.624 INFO:teuthology.orchestra.run.smithi186.stdout:3966 read 0x2e1c400 thru 0x2e2acd7 (0xe8d8 bytes) 2024-05-06T00:28:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:28:31.632 INFO:teuthology.orchestra.run.smithi186.stdout:3968 trunc from 0x58c2000 to 0x2a38200 2024-05-06T00:28:31.701 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:31.698+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:31.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:31.772 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:31.766+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:31.774 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:31.774 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:31.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:31.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:31.871 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:31.872 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:31.873+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:31.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:31.889 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:31.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:31.926 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:31.946 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:31.946 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:31.958 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:31.959 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:31.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:31.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:31.974 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:31.973+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:32.001 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 0, 5, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:32.001 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:32.002 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:28:32.074 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:32.070+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:32.098 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:32.098 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:32.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.639189689000006, 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:28:32.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:28:32.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.638387699999996, 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:32.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:28:32.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.638301245999997, 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.108 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:28:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:28:32.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:28:32.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.638228652999999, 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.112 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:32.116 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.638168801999996, 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:32.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.638093119000004, 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.121 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.630387545999994, 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.125 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:28:32.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.630333339000003, 2024-05-06T00:28:32.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:28:32.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:28:32.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.630281190000005, 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.134 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:32.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:28:32.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:28:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.628323475000002, 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.628247134999995, 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.813109428000004, 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.813056986999996, 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:28:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:28:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.806037004000004, 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:28:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:28:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:28:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.805990292999994, 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:28:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.805660184999994, 2024-05-06T00:28:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:28:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:28:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:28:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.805632263000007, 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:32.175 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:32.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.794765373999994, 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:28:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.1371155729999991, 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:28:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:28:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.1359309579999994, 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:32.193 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:32.195 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:32.190+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:32.195 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:32.295 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:32.293+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:32.311 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:32.311 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:32.311 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:32.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:28:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:28:32.348 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-05-06T00:28:31.613486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":741695799,"num_objects":1390,"num_object_clones":321,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1390,"num_whiteouts":728,"num_read":170652,"num_read_kb":506452,"num_write":11571,"num_write_kb":133057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3395,"num_bytes_recovered":36283961,"num_keys_recovered":624,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10793,"ondisk_log_size":10793,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":91,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":334588,"kb_used_data":76704,"kb_used_omap":9,"kb_used_meta":257782,"kb_avail":565896452,"statfs":{"total":579820584960,"available":579477966848,"internally_reserved":0,"allocated":78544896,"data_stored":99583010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":263969440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.634899"},"pg_stats":[{"pgid":"3.1f","version":"98'378","reported_seq":9651,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.964093+0000","last_change":"2024-05-06T00:28:26.984722+0000","last_active":"2024-05-06T00:28:26.984722+0000","last_peered":"2024-05-06T00:28:25.964093+0000","last_clean":"2024-05-06T00:28:25.964093+0000","last_became_active":"2024-05-06T00:28:18.922494+0000","last_became_peered":"2024-05-06T00:28:18.922494+0000","last_unstale":"2024-05-06T00:28:25.964093+0000","last_undegraded":"2024-05-06T00:28:25.964093+0000","last_fullsized":"2024-05-06T00:28:25.964093+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":12551303,"num_objects":40,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":21,"num_read":8603,"num_read_kb":30712,"num_write":334,"num_write_kb":1383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":499093,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"98'200","reported_seq":8400,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.982508+0000","last_change":"2024-05-06T00:28:26.981438+0000","last_active":"2024-05-06T00:28:26.981438+0000","last_peered":"2024-05-06T00:28:25.982508+0000","last_clean":"2024-05-06T00:28:25.982508+0000","last_became_active":"2024-05-06T00:28:18.921843+0000","last_became_peered":"2024-05-06T00:28:18.921843+0000","last_unstale":"2024-05-06T00:28:25.982508+0000","last_undegraded":"2024-05-06T00:28:25.982508+0000","last_fullsized":"2024-05-06T00:28:25.982508+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":59500063,"num_objects":53,"num_object_clones":15,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":23,"num_read":7925,"num_read_kb":40797,"num_write":179,"num_write_kb":16421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":5149215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"95'192","reported_seq":1220,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.984609+0000","last_change":"2024-05-06T00:28:24.969079+0000","last_active":"2024-05-06T00:28:26.984609+0000","last_peered":"2024-05-06T00:28:26.984609+0000","last_clean":"2024-05-06T00:28:26.984609+0000","last_became_active":"2024-05-06T00:28:24.967045+0000","last_became_peered":"2024-05-06T00:28:24.967045+0000","last_unstale":"2024-05-06T00:28:26.984609+0000","last_undegraded":"2024-05-06T00:28:26.984609+0000","last_fullsized":"2024-05-06T00:28:26.984609+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:35:35.593717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":13308928,"num_objects":40,"num_object_clones":9,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":26,"num_read":766,"num_read_kb":8516,"num_write":165,"num_write_kb":1529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"98'512","reported_seq":5953,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.995850+0000","last_change":"2024-05-06T00:28:26.994492+0000","last_active":"2024-05-06T00:28:26.995850+0000","last_peered":"2024-05-06T00:28:26.995850+0000","last_clean":"2024-05-06T00:28:26.995850+0000","last_became_active":"2024-05-06T00:28:26.992313+0000","last_became_peered":"2024-05-06T00:28:26.992313+0000","last_unstale":"2024-05-06T00:28:26.995850+0000","last_undegraded":"2024-05-06T00:28:26.995850+0000","last_fullsized":"2024-05-06T00:28:26.995850+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:49:20.404631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":21209653,"num_objects":44,"num_object_clones":12,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":21,"num_read":7412,"num_read_kb":24228,"num_write":480,"num_write_kb":4198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"98'747","reported_seq":5949,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.984902+0000","last_change":"2024-05-06T00:28:05.824053+0000","last_active":"2024-05-06T00:28:26.984902+0000","last_peered":"2024-05-06T00:28:26.984902+0000","last_clean":"2024-05-06T00:28:26.984902+0000","last_became_active":"2024-05-06T00:28:05.820651+0000","last_became_peered":"2024-05-06T00:28:05.820651+0000","last_unstale":"2024-05-06T00:28:26.984902+0000","last_undegraded":"2024-05-06T00:28:26.984902+0000","last_fullsized":"2024-05-06T00:28:26.984902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":747,"log_dups_size":0,"ondisk_log_size":747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:25:53.284956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":52195949,"num_objects":76,"num_object_clones":23,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":36,"num_read":7940,"num_read_kb":25235,"num_write":840,"num_write_kb":6871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"98'279","reported_seq":4501,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.990233+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:28:26.990233+0000","last_peered":"2024-05-06T00:28:26.990233+0000","last_clean":"2024-05-06T00:28:26.990233+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:28:26.990233+0000","last_undegraded":"2024-05-06T00:28:26.990233+0000","last_fullsized":"2024-05-06T00:28:26.990233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:21:46.061114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":31575721,"num_objects":42,"num_object_clones":7,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":19,"num_read":4890,"num_read_kb":20509,"num_write":320,"num_write_kb":13412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"95'526","reported_seq":2095,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.985568+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:28:26.985568+0000","last_peered":"2024-05-06T00:28:26.985568+0000","last_clean":"2024-05-06T00:28:26.985568+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:28:26.985568+0000","last_undegraded":"2024-05-06T00:28:26.985568+0000","last_fullsized":"2024-05-06T00:28:26.985568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:04:50.479784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":17373333,"num_objects":41,"num_object_clones":10,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":19,"num_read":3762,"num_read_kb":10471,"num_write":711,"num_write_kb":4559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"98'433","reported_seq":11809,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.980723+0000","last_change":"2024-05-06T00:28:06.811900+0000","last_active":"2024-05-06T00:28:26.980723+0000","last_peered":"2024-05-06T00:28:26.980723+0000","last_clean":"2024-05-06T00:28:26.980723+0000","last_became_active":"2024-05-06T00:28:06.811388+0000","last_became_peered":"2024-05-06T00:28:06.811388+0000","last_unstale":"2024-05-06T00:28:26.980723+0000","last_undegraded":"2024-05-06T00:28:26.980723+0000","last_fullsized":"2024-05-06T00:28:26.980723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:25:52.936851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":10638864,"num_objects":38,"num_object_clones":10,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":18,"num_read":11058,"num_read_kb":19066,"num_write":266,"num_write_kb":1487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":287776,"num_keys_recovered":35,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"94'198","reported_seq":2395,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.986577+0000","last_change":"2024-05-06T00:28:25.971398+0000","last_active":"2024-05-06T00:28:26.986577+0000","last_peered":"2024-05-06T00:28:26.986577+0000","last_clean":"2024-05-06T00:28:26.986577+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:28:26.986577+0000","last_undegraded":"2024-05-06T00:28:26.986577+0000","last_fullsized":"2024-05-06T00:28:26.986577+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:23:13.763244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":45496874,"num_objects":38,"num_object_clones":9,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":23,"num_read":1655,"num_read_kb":8438,"num_write":175,"num_write_kb":3582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"98'206","reported_seq":1606,"reported_epoch":102,"state":"unknown","last_fresh":"2024-05-06T00:28:26.985491+0000","last_change":"2024-05-06T00:28:27.512937+0000","last_active":"2024-05-06T00:28:26.985491+0000","last_peered":"2024-05-06T00:28:25.981586+0000","last_clean":"2024-05-06T00:28:25.981586+0000","last_became_active":"2024-05-06T00:28:25.973996+0000","last_became_peered":"2024-05-06T00:28:25.973996+0000","last_unstale":"2024-05-06T00:28:26.985491+0000","last_undegraded":"2024-05-06T00:28:26.985491+0000","last_fullsized":"2024-05-06T00:28:26.985491+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":43817027,"num_objects":47,"num_object_clones":14,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":25,"num_read":1371,"num_read_kb":11760,"num_write":219,"num_write_kb":1147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1037517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"95'157","reported_seq":4428,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.991030+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:28:26.991030+0000","last_peered":"2024-05-06T00:28:26.991030+0000","last_clean":"2024-05-06T00:28:26.991030+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:28:26.991030+0000","last_undegraded":"2024-05-06T00:28:26.991030+0000","last_fullsized":"2024-05-06T00:28:26.991030+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:15:21.396646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10971648,"num_objects":40,"num_object_clones":12,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":24,"num_read":4154,"num_read_kb":5525,"num_write":137,"num_write_kb":14663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"98'138","reported_seq":5259,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.972930+0000","last_change":"2024-05-06T00:28:26.991839+0000","last_active":"2024-05-06T00:28:26.991839+0000","last_peered":"2024-05-06T00:28:25.972930+0000","last_clean":"2024-05-06T00:28:25.972930+0000","last_became_active":"2024-05-06T00:28:24.969079+0000","last_became_peered":"2024-05-06T00:28:24.969079+0000","last_unstale":"2024-05-06T00:28:25.972930+0000","last_undegraded":"2024-05-06T00:28:25.972930+0000","last_fullsized":"2024-05-06T00:28:25.972930+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":19029097,"num_objects":35,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":18,"num_read":4749,"num_read_kb":12519,"num_write":111,"num_write_kb":1103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":1045536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.973780+0000","last_change":"2024-05-06T00:28:26.989466+0000","last_active":"2024-05-06T00:28:26.989466+0000","last_peered":"2024-05-06T00:28:25.973780+0000","last_clean":"2024-05-06T00:28:25.973780+0000","last_became_active":"2024-05-06T00:28:15.911172+0000","last_became_peered":"2024-05-06T00:28:15.911172+0000","last_unstale":"2024-05-06T00:28:25.973780+0000","last_undegraded":"2024-05-06T00:28:25.973780+0000","last_fullsized":"2024-05-06T00:28:25.973780+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'734","reported_seq":5049,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.985734+0000","last_change":"2024-05-06T00:28:06.817938+0000","last_active":"2024-05-06T00:28:26.985734+0000","last_peered":"2024-05-06T00:28:26.985734+0000","last_clean":"2024-05-06T00:28:26.985734+0000","last_became_active":"2024-05-06T00:28:06.816588+0000","last_became_peered":"2024-05-06T00:28:06.816588+0000","last_unstale":"2024-05-06T00:28:26.985734+0000","last_undegraded":"2024-05-06T00:28:26.985734+0000","last_fullsized":"2024-05-06T00:28:26.985734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:53:39.193632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":12535414,"num_objects":37,"num_object_clones":7,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":17,"num_read":8383,"num_read_kb":15540,"num_write":893,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":186,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.966128+0000","last_change":"2024-05-06T00:28:26.987793+0000","last_active":"2024-05-06T00:28:26.987793+0000","last_peered":"2024-05-06T00:28:25.966128+0000","last_clean":"2024-05-06T00:28:25.966128+0000","last_became_active":"2024-05-06T00:28:15.899763+0000","last_became_peered":"2024-05-06T00:28:15.899763+0000","last_unstale":"2024-05-06T00:28:25.966128+0000","last_undegraded":"2024-05-06T00:28:25.966128+0000","last_fullsized":"2024-05-06T00:28:25.966128+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"98'455","reported_seq":3409,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.982363+0000","last_change":"2024-05-06T00:26:45.714607+0000","last_active":"2024-05-06T00:28:26.982363+0000","last_peered":"2024-05-06T00:28:26.982363+0000","last_clean":"2024-05-06T00:28:26.982363+0000","last_became_active":"2024-05-06T00:26:43.684056+0000","last_became_peered":"2024-05-06T00:26:43.684056+0000","last_unstale":"2024-05-06T00:28:26.982363+0000","last_undegraded":"2024-05-06T00:28:26.982363+0000","last_fullsized":"2024-05-06T00:28:26.982363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:41:07.870346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":17168421,"num_objects":38,"num_object_clones":10,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":20,"num_read":5162,"num_read_kb":9714,"num_write":507,"num_write_kb":1995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.963577+0000","last_change":"2024-05-06T00:28:26.986139+0000","last_active":"2024-05-06T00:28:26.986139+0000","last_peered":"2024-05-06T00:28:25.963577+0000","last_clean":"2024-05-06T00:28:25.963577+0000","last_became_active":"2024-05-06T00:28:15.907864+0000","last_became_peered":"2024-05-06T00:28:15.907864+0000","last_unstale":"2024-05-06T00:28:25.963577+0000","last_undegraded":"2024-05-06T00:28:25.963577+0000","last_fullsized":"2024-05-06T00:28:25.963577+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"98'423","reported_seq":6726,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.986553+0000","last_change":"2024-05-06T00:26:45.725004+0000","last_active":"2024-05-06T00:28:26.986553+0000","last_peered":"2024-05-06T00:28:26.986553+0000","last_clean":"2024-05-06T00:28:26.986553+0000","last_became_active":"2024-05-06T00:26:43.689833+0000","last_became_peered":"2024-05-06T00:26:43.689833+0000","last_unstale":"2024-05-06T00:28:26.986553+0000","last_undegraded":"2024-05-06T00:28:26.986553+0000","last_fullsized":"2024-05-06T00:28:26.986553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:07:20.134965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":40517745,"num_objects":55,"num_object_clones":12,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":30,"num_read":7041,"num_read_kb":24303,"num_write":543,"num_write_kb":4790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":192,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.986659+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:28:26.986659+0000","last_peered":"2024-05-06T00:28:26.986659+0000","last_clean":"2024-05-06T00:28:26.986659+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:28:26.986659+0000","last_undegraded":"2024-05-06T00:28:26.986659+0000","last_fullsized":"2024-05-06T00:28:26.986659+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"83'238","reported_seq":1575,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.984332+0000","last_change":"2024-05-06T00:28:24.969115+0000","last_active":"2024-05-06T00:28:26.984332+0000","last_peered":"2024-05-06T00:28:26.984332+0000","last_clean":"2024-05-06T00:28:26.984332+0000","last_became_active":"2024-05-06T00:28:24.967511+0000","last_became_peered":"2024-05-06T00:28:24.967511+0000","last_unstale":"2024-05-06T00:28:26.984332+0000","last_undegraded":"2024-05-06T00:28:26.984332+0000","last_fullsized":"2024-05-06T00:28:26.984332+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:29:38.326336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":3608645,"num_objects":30,"num_object_clones":4,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":16,"num_read":1940,"num_read_kb":6961,"num_write":257,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":432741,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"94'120","reported_seq":1300,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.990398+0000","last_change":"2024-05-06T00:28:02.884939+0000","last_active":"2024-05-06T00:28:26.990398+0000","last_peered":"2024-05-06T00:28:26.990398+0000","last_clean":"2024-05-06T00:28:26.990398+0000","last_became_active":"2024-05-06T00:28:02.782051+0000","last_became_peered":"2024-05-06T00:28:02.782051+0000","last_unstale":"2024-05-06T00:28:26.990398+0000","last_undegraded":"2024-05-06T00:28:26.990398+0000","last_fullsized":"2024-05-06T00:28:26.990398+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:09:15.030437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":5513819,"num_objects":31,"num_object_clones":6,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":19,"num_read":830,"num_read_kb":7751,"num_write":101,"num_write_kb":573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.983900+0000","last_change":"2024-05-06T00:28:21.627071+0000","last_active":"2024-05-06T00:28:26.983900+0000","last_peered":"2024-05-06T00:28:26.983900+0000","last_clean":"2024-05-06T00:28:26.983900+0000","last_became_active":"2024-05-06T00:28:21.626670+0000","last_became_peered":"2024-05-06T00:28:21.626670+0000","last_unstale":"2024-05-06T00:28:26.983900+0000","last_undegraded":"2024-05-06T00:28:26.983900+0000","last_fullsized":"2024-05-06T00:28:26.983900+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:03:33.986927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"98'400","reported_seq":3365,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.984787+0000","last_change":"2024-05-06T00:26:45.725833+0000","last_active":"2024-05-06T00:28:26.984787+0000","last_peered":"2024-05-06T00:28:26.984787+0000","last_clean":"2024-05-06T00:28:26.984787+0000","last_became_active":"2024-05-06T00:26:43.690046+0000","last_became_peered":"2024-05-06T00:26:43.690046+0000","last_unstale":"2024-05-06T00:28:26.984787+0000","last_undegraded":"2024-05-06T00:28:26.984787+0000","last_fullsized":"2024-05-06T00:28:26.984787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:01:29.749344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":12213382,"num_objects":38,"num_object_clones":6,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":18,"num_read":4107,"num_read_kb":15971,"num_write":575,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":182,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.980829+0000","last_change":"2024-05-06T00:26:35.647778+0000","last_active":"2024-05-06T00:28:26.980829+0000","last_peered":"2024-05-06T00:28:26.980829+0000","last_clean":"2024-05-06T00:28:26.980829+0000","last_became_active":"2024-05-06T00:26:35.647314+0000","last_became_peered":"2024-05-06T00:26:35.647314+0000","last_unstale":"2024-05-06T00:28:26.980829+0000","last_undegraded":"2024-05-06T00:28:26.980829+0000","last_fullsized":"2024-05-06T00:28:26.980829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:27:58.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'182","reported_seq":3301,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.985662+0000","last_change":"2024-05-06T00:28:05.821874+0000","last_active":"2024-05-06T00:28:26.985662+0000","last_peered":"2024-05-06T00:28:26.985662+0000","last_clean":"2024-05-06T00:28:26.985662+0000","last_became_active":"2024-05-06T00:28:05.820367+0000","last_became_peered":"2024-05-06T00:28:05.820367+0000","last_unstale":"2024-05-06T00:28:26.985662+0000","last_undegraded":"2024-05-06T00:28:26.985662+0000","last_fullsized":"2024-05-06T00:28:26.985662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:58:45.867520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":33550848,"num_objects":45,"num_object_clones":11,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":22,"num_read":2677,"num_read_kb":10252,"num_write":169,"num_write_kb":1736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":125,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.964887+0000","last_change":"2024-05-06T00:28:26.985355+0000","last_active":"2024-05-06T00:28:26.985355+0000","last_peered":"2024-05-06T00:28:25.964887+0000","last_clean":"2024-05-06T00:28:25.964887+0000","last_became_active":"2024-05-06T00:28:15.905631+0000","last_became_peered":"2024-05-06T00:28:15.905631+0000","last_unstale":"2024-05-06T00:28:25.964887+0000","last_undegraded":"2024-05-06T00:28:25.964887+0000","last_fullsized":"2024-05-06T00:28:25.964887+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"94'267","reported_seq":7347,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.986951+0000","last_change":"2024-05-06T00:28:25.970874+0000","last_active":"2024-05-06T00:28:26.986951+0000","last_peered":"2024-05-06T00:28:26.986951+0000","last_clean":"2024-05-06T00:28:26.986951+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:28:26.986951+0000","last_undegraded":"2024-05-06T00:28:26.986951+0000","last_fullsized":"2024-05-06T00:28:26.986951+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:11:09.577538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":12728394,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":16,"num_read":7756,"num_read_kb":15434,"num_write":197,"num_write_kb":1413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.992668+0000","last_change":"2024-05-06T00:28:21.624595+0000","last_active":"2024-05-06T00:28:26.992668+0000","last_peered":"2024-05-06T00:28:26.992668+0000","last_clean":"2024-05-06T00:28:26.992668+0000","last_became_active":"2024-05-06T00:28:21.624095+0000","last_became_peered":"2024-05-06T00:28:21.624095+0000","last_unstale":"2024-05-06T00:28:26.992668+0000","last_undegraded":"2024-05-06T00:28:26.992668+0000","last_fullsized":"2024-05-06T00:28:26.992668+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:42:42.260726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":68,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.963244+0000","last_change":"2024-05-06T00:28:26.985772+0000","last_active":"2024-05-06T00:28:26.985772+0000","last_peered":"2024-05-06T00:28:25.963244+0000","last_clean":"2024-05-06T00:28:25.963244+0000","last_became_active":"2024-05-06T00:28:15.905648+0000","last_became_peered":"2024-05-06T00:28:15.905648+0000","last_unstale":"2024-05-06T00:28:25.963244+0000","last_undegraded":"2024-05-06T00:28:25.963244+0000","last_fullsized":"2024-05-06T00:28:25.963244+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"98'359","reported_seq":2092,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.980832+0000","last_change":"2024-05-06T00:26:45.714303+0000","last_active":"2024-05-06T00:28:26.980832+0000","last_peered":"2024-05-06T00:28:26.980832+0000","last_clean":"2024-05-06T00:28:26.980832+0000","last_became_active":"2024-05-06T00:26:43.689921+0000","last_became_peered":"2024-05-06T00:26:43.689921+0000","last_unstale":"2024-05-06T00:28:26.980832+0000","last_undegraded":"2024-05-06T00:28:26.980832+0000","last_fullsized":"2024-05-06T00:28:26.980832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:06:37.666022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":26317992,"num_objects":49,"num_object_clones":12,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":26,"num_read":2618,"num_read_kb":6604,"num_write":431,"num_write_kb":2184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"98'397","reported_seq":4180,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.988417+0000","last_change":"2024-05-06T00:26:45.721816+0000","last_active":"2024-05-06T00:28:26.988417+0000","last_peered":"2024-05-06T00:28:26.988417+0000","last_clean":"2024-05-06T00:28:26.988417+0000","last_became_active":"2024-05-06T00:26:43.709213+0000","last_became_peered":"2024-05-06T00:26:43.709213+0000","last_unstale":"2024-05-06T00:28:26.988417+0000","last_undegraded":"2024-05-06T00:28:26.988417+0000","last_fullsized":"2024-05-06T00:28:26.988417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:36:59.315180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":21510894,"num_objects":49,"num_object_clones":11,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":27,"num_read":4668,"num_read_kb":19968,"num_write":535,"num_write_kb":11754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"98'379","reported_seq":4704,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.971616+0000","last_change":"2024-05-06T00:28:26.989148+0000","last_active":"2024-05-06T00:28:26.989148+0000","last_peered":"2024-05-06T00:28:25.971616+0000","last_clean":"2024-05-06T00:28:25.971616+0000","last_became_active":"2024-05-06T00:28:18.918651+0000","last_became_peered":"2024-05-06T00:28:18.918651+0000","last_unstale":"2024-05-06T00:28:25.971616+0000","last_undegraded":"2024-05-06T00:28:25.971616+0000","last_fullsized":"2024-05-06T00:28:25.971616+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":23911174,"num_objects":59,"num_object_clones":14,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":31,"num_read":5563,"num_read_kb":25185,"num_write":275,"num_write_kb":4305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":498438,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"98'213","reported_seq":5486,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.986268+0000","last_change":"2024-05-06T00:28:06.813891+0000","last_active":"2024-05-06T00:28:26.986268+0000","last_peered":"2024-05-06T00:28:26.986268+0000","last_clean":"2024-05-06T00:28:26.986268+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:28:26.986268+0000","last_undegraded":"2024-05-06T00:28:26.986268+0000","last_fullsized":"2024-05-06T00:28:26.986268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:19:50.458298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":17206518,"num_objects":39,"num_object_clones":8,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":22,"num_read":3861,"num_read_kb":16180,"num_write":227,"num_write_kb":5573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"89'263","reported_seq":2394,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.986179+0000","last_change":"2024-05-06T00:28:25.977493+0000","last_active":"2024-05-06T00:28:26.986179+0000","last_peered":"2024-05-06T00:28:26.986179+0000","last_clean":"2024-05-06T00:28:26.986179+0000","last_became_active":"2024-05-06T00:28:25.973644+0000","last_became_peered":"2024-05-06T00:28:25.973644+0000","last_unstale":"2024-05-06T00:28:26.986179+0000","last_undegraded":"2024-05-06T00:28:26.986179+0000","last_fullsized":"2024-05-06T00:28:26.986179+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:02:38.380565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":13230166,"num_objects":40,"num_object_clones":5,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":25,"num_read":2051,"num_read_kb":7545,"num_write":217,"num_write_kb":1459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"98'666","reported_seq":8787,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.982573+0000","last_change":"2024-05-06T00:28:26.982789+0000","last_active":"2024-05-06T00:28:26.982789+0000","last_peered":"2024-05-06T00:28:25.982573+0000","last_clean":"2024-05-06T00:28:25.982573+0000","last_became_active":"2024-05-06T00:28:18.913184+0000","last_became_peered":"2024-05-06T00:28:18.913184+0000","last_unstale":"2024-05-06T00:28:25.982573+0000","last_undegraded":"2024-05-06T00:28:25.982573+0000","last_fullsized":"2024-05-06T00:28:25.982573+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":16950345,"num_objects":55,"num_object_clones":13,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":33,"num_read":10141,"num_read_kb":18369,"num_write":998,"num_write_kb":2746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":378953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"98'369","reported_seq":9329,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.985283+0000","last_change":"2024-05-06T00:28:25.980783+0000","last_active":"2024-05-06T00:28:26.985283+0000","last_peered":"2024-05-06T00:28:26.985283+0000","last_clean":"2024-05-06T00:28:26.985283+0000","last_became_active":"2024-05-06T00:28:25.975566+0000","last_became_peered":"2024-05-06T00:28:25.975566+0000","last_unstale":"2024-05-06T00:28:26.985283+0000","last_undegraded":"2024-05-06T00:28:26.985283+0000","last_fullsized":"2024-05-06T00:28:26.985283+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:25:04.853981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":22849678,"num_objects":48,"num_object_clones":12,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":24,"num_read":10301,"num_read_kb":15826,"num_write":465,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"98'238","reported_seq":2776,"reported_epoch":102,"state":"unknown","last_fresh":"2024-05-06T00:28:26.985748+0000","last_change":"2024-05-06T00:28:27.512960+0000","last_active":"2024-05-06T00:28:26.985748+0000","last_peered":"2024-05-06T00:28:25.967882+0000","last_clean":"2024-05-06T00:28:25.967882+0000","last_became_active":"2024-05-06T00:28:24.967157+0000","last_became_peered":"2024-05-06T00:28:24.967157+0000","last_unstale":"2024-05-06T00:28:26.985748+0000","last_undegraded":"2024-05-06T00:28:26.985748+0000","last_fullsized":"2024-05-06T00:28:26.985748+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":13018579,"num_objects":37,"num_object_clones":8,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":19,"num_read":2636,"num_read_kb":12646,"num_write":291,"num_write_kb":1375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":966177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"98'367","reported_seq":9540,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-06T00:28:26.983999+0000","last_change":"2024-05-06T00:28:25.970188+0000","last_active":"2024-05-06T00:28:26.983999+0000","last_peered":"2024-05-06T00:28:26.983999+0000","last_clean":"2024-05-06T00:28:26.983999+0000","last_became_active":"2024-05-06T00:28:25.969191+0000","last_became_peered":"2024-05-06T00:28:25.969191+0000","last_unstale":"2024-05-06T00:28:26.983999+0000","last_undegraded":"2024-05-06T00:28:26.983999+0000","last_fullsized":"2024-05-06T00:28:26.983999+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:38:08.596022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":29518864,"num_objects":40,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":19,"num_read":10268,"num_read_kb":15815,"num_write":224,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"82'169","reported_seq":4085,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.971228+0000","last_change":"2024-05-06T00:28:26.987496+0000","last_active":"2024-05-06T00:28:26.987496+0000","last_peered":"2024-05-06T00:28:25.971228+0000","last_clean":"2024-05-06T00:28:25.971228+0000","last_became_active":"2024-05-06T00:28:18.915237+0000","last_became_peered":"2024-05-06T00:28:18.915237+0000","last_unstale":"2024-05-06T00:28:25.971228+0000","last_undegraded":"2024-05-06T00:28:25.971228+0000","last_fullsized":"2024-05-06T00:28:25.971228+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":23233580,"num_objects":43,"num_object_clones":8,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":27,"num_read":3752,"num_read_kb":6997,"num_write":190,"num_write_kb":1775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":528428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"89'213","reported_seq":6351,"reported_epoch":101,"state":"peering","last_fresh":"2024-05-06T00:28:25.963233+0000","last_change":"2024-05-06T00:28:26.985313+0000","last_active":"2024-05-06T00:28:26.985313+0000","last_peered":"2024-05-06T00:28:25.963233+0000","last_clean":"2024-05-06T00:28:25.963233+0000","last_became_active":"2024-05-06T00:28:18.921670+0000","last_became_peered":"2024-05-06T00:28:18.921670+0000","last_unstale":"2024-05-06T00:28:25.963233+0000","last_undegraded":"2024-05-06T00:28:25.963233+0000","last_fullsized":"2024-05-06T00:28:25.963233+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":31747072,"num_objects":55,"num_object_clones":16,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":29,"num_read":5798,"num_read_kb":14341,"num_write":208,"num_write_kb":2313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1726976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"92'303","reported_seq":6448,"reported_epoch":102,"state":"unknown","last_fresh":"2024-05-06T00:28:26.984935+0000","last_change":"2024-05-06T00:28:27.512834+0000","last_active":"2024-05-06T00:28:26.984935+0000","last_peered":"2024-05-06T00:28:25.966586+0000","last_clean":"2024-05-06T00:28:25.966586+0000","last_became_active":"2024-05-06T00:28:24.967707+0000","last_became_peered":"2024-05-06T00:28:24.967707+0000","last_unstale":"2024-05-06T00:28:26.984935+0000","last_undegraded":"2024-05-06T00:28:26.984935+0000","last_fullsized":"2024-05-06T00:28:26.984935+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":26105441,"num_objects":36,"num_object_clones":8,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":15,"num_read":6748,"num_read_kb":23219,"num_write":230,"num_write_kb":3515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":6650147,"num_keys_recovered":69,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":741105431,"num_objects":1388,"num_object_clones":321,"num_object_copies":2776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":728,"num_read":170586,"num_read_kb":506397,"num_write":11470,"num_write_kb":131429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3389,"num_bytes_recovered":34512857,"num_keys_recovered":624,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":82538496,"data_stored":111764411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10721,"ondisk_log_size":10721,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640285,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49520,"kb_used_data":10012,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94322320,"statfs":{"total":96636764160,"available":96586055680,"internally_reserved":0,"allocated":10252288,"data_stored":11561912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640283,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67208,"kb_used_data":17980,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94304632,"statfs":{"total":96636764160,"available":96567943168,"internally_reserved":0,"allocated":18411520,"data_stored":25436675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640283,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40680,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331160,"statfs":{"total":96636764160,"available":96595107840,"internally_reserved":0,"allocated":1204224,"data_stored":495090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640284,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64600,"kb_used_data":15368,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94307240,"statfs":{"total":96636764160,"available":96570613760,"internally_reserved":0,"allocated":15736832,"data_stored":18734153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640285,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70848,"kb_used_data":21620,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94300992,"statfs":{"total":96636764160,"available":96564215808,"internally_reserved":0,"allocated":22138880,"data_stored":31360956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640283,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":10548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":10801152,"data_stored":11994224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9015296,"data_stored":10914138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20340736,"data_stored":30275498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14532608,"data_stored":18239063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12382208,"data_stored":16321933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17207296,"data_stored":24941585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9060352,"data_stored":11072194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:28:32.396 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:32.397+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:32.447 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:32.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:32.447 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:32.447 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:32.447 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:32.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:32.497 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:32.494+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:32.597 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:32.597+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:32.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:32.634 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:32.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:32.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:32.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.634 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 93.827508211999998, 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:28:32.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:28:32.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:28:32.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:28:32.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:28:32.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 93.827492008999997, 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.643 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:28:32.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:28:32.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:28:32.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:28:32.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:28:32.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:28:32.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:28:32.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.650 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34113 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 845824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.819361926999999, 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.073299384, 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34113 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2200000000000004e-07 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812509+0000", 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-06 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812519+0000", 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6979999999999993e-06 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812528+0000", 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5370000000000003e-06 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817889+0000", 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053609809999999999 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817903+0000", 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3545e-05 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885659+0000", 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.067755797 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885734+0000", 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5698999999999995e-05 2024-05-06T00:28:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885805+0000", 2024-05-06T00:28:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0442000000000001e-05 2024-05-06T00:28:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34114 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 851968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.819329890999999, 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0734813590000001, 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34114 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-07 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812540+0000", 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4509999999999999e-06 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5069999999999997e-06 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812552+0000", 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817936+0000", 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053840650000000004 2024-05-06T00:28:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817944+0000", 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5890000000000003e-06 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885879+0000", 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0679342119999999 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885952+0000", 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2924999999999997e-05 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886019+0000", 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7358000000000001e-05 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34115 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 858112~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.819294782, 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0736428419999999, 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34115 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9700000000000002e-07 2024-05-06T00:28:32.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812575+0000", 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.402e-06 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812587+0000", 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7979999999999996e-06 2024-05-06T00:28:32.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817971+0000", 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053842719999999998 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817979+0000", 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4250000000000001e-06 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886063+0000", 2024-05-06T00:28:32.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068084475 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886129+0000", 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6109e-05 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886216+0000", 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6312999999999998e-05 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.819261791999999, 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:32.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:28:32.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:28:32.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.819230422, 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:32.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:32.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:28:32.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:32.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:32.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:28:32.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:28:32.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:32.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:28:32.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:28:32.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:28:32.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:28:32.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.819196633000001, 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.684 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:28:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.819016177000002, 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818982534, 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:28:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818958752, 2024-05-06T00:28:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:28:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818932475, 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:28:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:28:32.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:28:32.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818889234, 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:28:32.711 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818859605, 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.716 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:28:32.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:28:32.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:28:32.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.722 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818826755, 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:28:32.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818795117000001, 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:32.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:28:32.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818762315000001, 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:28:32.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:28:32.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818730282000001, 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:28:32.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:32.744 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818698203, 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:28:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:28:32.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:28:32.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.818666895, 2024-05-06T00:28:32.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:28:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:28:32.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:32.753 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:32.754 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:32.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:32.756 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:32.753+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:32.758 INFO:teuthology.orchestra.run.smithi186.stdout:3969 write 0x3a18000 thru 0x3a1f79d (0x779e bytes) 2024-05-06T00:28:32.758 INFO:teuthology.orchestra.run.smithi186.stdout:3970 read 0x929800 thru 0x92fef2 (0x66f3 bytes) 2024-05-06T00:28:32.758 INFO:teuthology.orchestra.run.smithi186.stdout:3972 read 0x37e8800 thru 0x37eae77 (0x2678 bytes) 2024-05-06T00:28:32.758 INFO:teuthology.orchestra.run.smithi186.stdout:3975 read 0x2afe00 thru 0x2b5d12 (0x5f13 bytes) 2024-05-06T00:28:32.758 INFO:teuthology.orchestra.run.smithi186.stdout:3977 read 0x31ee400 thru 0x31f08b2 (0x24b3 bytes) 2024-05-06T00:28:32.758 INFO:teuthology.orchestra.run.smithi186.stdout:3978 read 0x260c600 thru 0x2610f27 (0x4928 bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3979 read 0x2f81c00 thru 0x2f8b7ef (0x9bf0 bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3980 punch from 0x1c55000 to 0x1c5e600, (0x9600 bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3981 write 0xc7eee00 thru 0xc7fd2fd (0xe4fe bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3982 write 0x2159800 thru 0x215e5dc (0x4ddd bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3983 read 0x8746000 thru 0x87500cc (0xa0cd bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3985 write 0x2b800 thru 0x2bdc0 (0x5c1 bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3986 read 0x9385000 thru 0x938d037 (0x8038 bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3988 read 0x11a5e00 thru 0x11aa010 (0x4211 bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3991 write 0xb43fc00 thru 0xb44ec33 (0xf034 bytes) 2024-05-06T00:28:32.759 INFO:teuthology.orchestra.run.smithi186.stdout:3992 trunc from 0xc7fd400 to 0x9b2c00 2024-05-06T00:28:32.829 INFO:teuthology.orchestra.run.smithi186.stdout:3993 punch from 0x41a400 to 0x421600, (0x7200 bytes) 2024-05-06T00:28:32.851 INFO:teuthology.orchestra.run.smithi186.stdout:3995 write 0xe8f400 thru 0xe9916f (0x9d70 bytes) 2024-05-06T00:28:32.856 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:32.850+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:32.897 INFO:teuthology.orchestra.run.smithi186.stdout:4000 write 0x7d5aa00 thru 0x7d5ed74 (0x4375 bytes) 2024-05-06T00:28:32.901 INFO:teuthology.orchestra.run.smithi186.stdout:4002 write 0x3809c00 thru 0x3818883 (0xec84 bytes) 2024-05-06T00:28:32.901 INFO:teuthology.orchestra.run.smithi186.stdout:4006 read 0x3aadc00 thru 0x3ab0626 (0x2a27 bytes) 2024-05-06T00:28:32.938 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:32.939 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:32.939 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:32.947 INFO:teuthology.orchestra.run.smithi186.stdout:4007 write 0xd278600 thru 0xd27b20d (0x2c0e bytes) 2024-05-06T00:28:32.950 INFO:teuthology.orchestra.run.smithi186.stdout:4008 write 0xa81c600 thru 0xa81fd57 (0x3758 bytes) 2024-05-06T00:28:32.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:32.970 INFO:teuthology.orchestra.run.smithi186.stdout:4009 trunc from 0xd27b400 to 0x10c9800 2024-05-06T00:28:33.000 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:33.001+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:33.013 INFO:teuthology.orchestra.run.smithi186.stdout:4014 punch from 0x43f000 to 0x444e00, (0x5e00 bytes) 2024-05-06T00:28:33.015 INFO:teuthology.orchestra.run.smithi186.stdout:4018 read 0xe46c00 thru 0xe4f35b (0x875c bytes) 2024-05-06T00:28:33.016 INFO:teuthology.orchestra.run.smithi186.stdout:4020 punch from 0xcdde00 to 0xce0a00, (0x2c00 bytes) 2024-05-06T00:28:33.023 INFO:teuthology.orchestra.run.smithi186.stdout:4021 trunc from 0x10c9800 to 0x62600 2024-05-06T00:28:33.057 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:33.057+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:33.082 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:33.083 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:33.097 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:33.132 INFO:teuthology.orchestra.run.smithi186.stdout:4023 write 0xc426400 thru 0xc42c8ef (0x64f0 bytes) 2024-05-06T00:28:33.150 INFO:teuthology.orchestra.run.smithi186.stdout:4024 trunc from 0xc42ca00 to 0x53ac600 2024-05-06T00:28:33.184 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:33.185+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:33.218 INFO:teuthology.orchestra.run.smithi186.stdout:4025 write 0xbec6800 thru 0xbed3be2 (0xd3e3 bytes) 2024-05-06T00:28:33.239 INFO:teuthology.orchestra.run.smithi186.stdout:4026 trunc from 0xbed3c00 to 0x3a25a00 2024-05-06T00:28:33.259 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:33.257+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:33.275 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21566 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 417792~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247335984999999, 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090303018999999998, 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21566 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5400000000000002e-07 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026960+0000", 2024-05-06T00:28:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4449999999999999e-06 2024-05-06T00:28:33.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026966+0000", 2024-05-06T00:28:33.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6200000000000001e-06 2024-05-06T00:28:33.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026978+0000", 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1248e-05 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116885+0000", 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.089907163999999998 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116986+0000", 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101363 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117255+0000", 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026882499999999999 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21568 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 423936~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247228499999999, 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090403081999999996, 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.280 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21568 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027060+0000", 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027063+0000", 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3249999999999999e-06 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027072+0000", 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2859999999999999e-06 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027078+0000", 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117286+0000", 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090207976999999995 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117372+0000", 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5865000000000005e-05 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117463+0000", 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0662999999999996e-05 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21570 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 430080~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:33.284 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247201891, 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090545366000000002, 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21570 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027087+0000", 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3799999999999998e-07 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027089+0000", 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.12e-06 2024-05-06T00:28:33.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027095+0000", 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4590000000000003e-06 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027100+0000", 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117492+0000", 2024-05-06T00:28:33.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090391632999999999 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117546+0000", 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4177e-05 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117631+0000", 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5173999999999995e-05 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21572 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 436224~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.24717953, 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090693918999999998, 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21572 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027109+0000", 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7100000000000002e-07 2024-05-06T00:28:33.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027112+0000", 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5490000000000001e-06 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027119+0000", 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7829999999999999e-06 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027124+0000", 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.04e-06 2024-05-06T00:28:33.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117662+0000", 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090537113000000002 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117717+0000", 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5798999999999999e-05 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117802+0000", 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5063999999999997e-05 2024-05-06T00:28:33.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247155463999999, 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247134675000002, 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:28:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:28:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247114613000001, 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247095337000001, 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:28:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247075218999999, 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:28:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:28:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247043810000001, 2024-05-06T00:28:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:28:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:28:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.247010897999999, 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:28:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.246990311000001, 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.246970931, 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:28:33.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:33.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.246926250000001, 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.336 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.246861159000002, 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:28:33.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:33.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:28:33.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.246829585, 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:28:33.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:28:33.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.246807323999999, 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:28:33.345 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.349 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.246786208, 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.589069582, 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:28:33.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:28:33.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:28:33.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:28:33.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.58865934, 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:28:33.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.362 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:33.363 INFO:teuthology.orchestra.run.smithi186.stdout:4028 read 0x780800 thru 0x783a89 (0x328a bytes) 2024-05-06T00:28:33.364 INFO:teuthology.orchestra.run.smithi186.stdout:4030 write 0xc8b6200 thru 0xc8bc237 (0x6038 bytes) 2024-05-06T00:28:33.364 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:33.365 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:33.358+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:33.370 INFO:teuthology.orchestra.run.smithi186.stdout:4032 write 0xe31b400 thru 0xe325308 (0x9f09 bytes) 2024-05-06T00:28:33.372 INFO:teuthology.orchestra.run.smithi186.stdout:4033 read 0x3d79400 thru 0x3d7cc98 (0x3899 bytes) 2024-05-06T00:28:33.372 INFO:teuthology.orchestra.run.smithi186.stdout:4037 read 0xaf41a00 thru 0xaf4f634 (0xdc35 bytes) 2024-05-06T00:28:33.372 INFO:teuthology.orchestra.run.smithi186.stdout:4038 write 0xf93a00 thru 0xf950cf (0x16d0 bytes) 2024-05-06T00:28:33.373 INFO:teuthology.orchestra.run.smithi186.stdout:4040 write 0xcf51800 thru 0xcf53b74 (0x2375 bytes) 2024-05-06T00:28:33.373 INFO:teuthology.orchestra.run.smithi186.stdout:4042 write 0x935e000 thru 0x936282b (0x482c bytes) 2024-05-06T00:28:33.373 INFO:teuthology.orchestra.run.smithi186.stdout:4043 read 0x8a64200 thru 0x8a654de (0x12df bytes) 2024-05-06T00:28:33.373 INFO:teuthology.orchestra.run.smithi186.stdout:4045 read 0x2c7400 thru 0x2cc6c6 (0x52c7 bytes) 2024-05-06T00:28:33.373 INFO:teuthology.orchestra.run.smithi186.stdout:4046 read 0xda41200 thru 0xda4b86b (0xa66c bytes) 2024-05-06T00:28:33.373 INFO:teuthology.orchestra.run.smithi186.stdout:4047 read 0xc9c2800 thru 0xc9ca9bd (0x81be bytes) 2024-05-06T00:28:33.394 INFO:teuthology.orchestra.run.smithi186.stdout:4048 trunc from 0xe325400 to 0x24ea600 2024-05-06T00:28:33.447 INFO:teuthology.orchestra.run.smithi186.stdout:4049 read 0x581e00 thru 0x584050 (0x2251 bytes) 2024-05-06T00:28:33.451 INFO:teuthology.orchestra.run.smithi186.stdout:4051 punch from 0x1633200 to 0x163e000, (0xae00 bytes) 2024-05-06T00:28:33.451 INFO:teuthology.orchestra.run.smithi186.stdout:4055 read 0x2e4600 thru 0x2ef6ff (0xb100 bytes) 2024-05-06T00:28:33.464 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:33.458+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:33.471 INFO:teuthology.orchestra.run.smithi186.stdout:4056 trunc from 0x24ea600 to 0xae08200 2024-05-06T00:28:33.492 INFO:teuthology.orchestra.run.smithi186.stdout:4063 read 0x96af000 thru 0x96b59cf (0x69d0 bytes) 2024-05-06T00:28:33.494 INFO:teuthology.orchestra.run.smithi186.stdout:4064 read 0xb5d200 thru 0xb60b82 (0x3983 bytes) 2024-05-06T00:28:33.494 INFO:teuthology.orchestra.run.smithi186.stdout:4065 read 0x91a3400 thru 0x91b0354 (0xcf55 bytes) 2024-05-06T00:28:33.494 INFO:teuthology.orchestra.run.smithi186.stdout:4066 read 0xb78400 thru 0xb8499c (0xc59d bytes) 2024-05-06T00:28:33.495 INFO:teuthology.orchestra.run.smithi186.stdout:4068 punch from 0x5751e00 to 0x5760600, (0xe800 bytes) 2024-05-06T00:28:33.495 INFO:teuthology.orchestra.run.smithi186.stdout:4069 read 0x8b9ae00 thru 0x8ba56fc (0xa8fd bytes) 2024-05-06T00:28:33.514 INFO:teuthology.orchestra.run.smithi186.stdout:4070 trunc from 0xae08200 to 0x8a3200 2024-05-06T00:28:33.553 INFO:teuthology.orchestra.run.smithi186.stdout:4072 read 0xe3000 thru 0xedf3c (0xaf3d bytes) 2024-05-06T00:28:33.559 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:33.559 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:33.565 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:33.562+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:33.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:33.596 INFO:teuthology.orchestra.run.smithi186.stdout:4075 write 0xba39a00 thru 0xba3dede (0x44df bytes) 2024-05-06T00:28:33.598 INFO:teuthology.orchestra.run.smithi186.stdout:4077 punch from 0x50d3400 to 0x50d7c00, (0x4800 bytes) 2024-05-06T00:28:33.599 INFO:teuthology.orchestra.run.smithi186.stdout:4080 read 0xac0f400 thru 0xac121f6 (0x2df7 bytes) 2024-05-06T00:28:33.599 INFO:teuthology.orchestra.run.smithi186.stdout:4081 punch from 0xaeb2600 to 0xaec1600, (0xf000 bytes) 2024-05-06T00:28:33.599 INFO:teuthology.orchestra.run.smithi186.stdout:4083 write 0x20c00 thru 0x2491d (0x3d1e bytes) 2024-05-06T00:28:33.601 INFO:teuthology.orchestra.run.smithi186.stdout:4084 write 0x8587800 thru 0x8594e22 (0xd623 bytes) 2024-05-06T00:28:33.602 INFO:teuthology.orchestra.run.smithi186.stdout:4085 punch from 0x863ec00 to 0x8645600, (0x6a00 bytes) 2024-05-06T00:28:33.602 INFO:teuthology.orchestra.run.smithi186.stdout:4086 read 0xb4af400 thru 0xb4b6dff (0x7a00 bytes) 2024-05-06T00:28:33.603 INFO:teuthology.orchestra.run.smithi186.stdout:4091 write 0x5fdc200 thru 0x5fe8e64 (0xcc65 bytes) 2024-05-06T00:28:33.607 INFO:teuthology.orchestra.run.smithi186.stdout:4093 trunc from 0xba3e000 to 0xda6fc00 2024-05-06T00:28:33.642 INFO:teuthology.orchestra.run.smithi186.stdout:4094 punch from 0xc06e400 to 0xc07b800, (0xd400 bytes) 2024-05-06T00:28:33.642 INFO:teuthology.orchestra.run.smithi186.stdout:4096 read 0x3b4e600 thru 0x3b4e67c (0x7d bytes) 2024-05-06T00:28:33.642 INFO:teuthology.orchestra.run.smithi186.stdout:4097 read 0x36f6000 thru 0x37043cb (0xe3cc bytes) 2024-05-06T00:28:33.643 INFO:teuthology.orchestra.run.smithi186.stdout:4098 write 0x6839a00 thru 0x683c071 (0x2672 bytes) 2024-05-06T00:28:33.643 INFO:teuthology.orchestra.run.smithi186.stdout:4099 read 0x3e96e00 thru 0x3e9ba71 (0x4c72 bytes) 2024-05-06T00:28:33.643 INFO:teuthology.orchestra.run.smithi186.stdout:4102 read 0x2003e00 thru 0x200b7f2 (0x79f3 bytes) 2024-05-06T00:28:33.643 INFO:teuthology.orchestra.run.smithi186.stdout:4103 punch from 0xd9f4a00 to 0xda04200, (0xf800 bytes) 2024-05-06T00:28:33.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:28:33.676 INFO:teuthology.orchestra.run.smithi186.stdout:4106 write 0xe566e00 thru 0xe5721fa (0xb3fb bytes) 2024-05-06T00:28:33.679 INFO:teuthology.orchestra.run.smithi186.stdout:4109 read 0x238e00 thru 0x246d71 (0xdf72 bytes) 2024-05-06T00:28:33.682 INFO:teuthology.orchestra.run.smithi186.stdout:4110 read 0x23ed800 thru 0x23ee58c (0xd8d bytes) 2024-05-06T00:28:33.682 INFO:teuthology.orchestra.run.smithi186.stdout:4117 punch from 0xbb61c00 to 0xbb6c800, (0xac00 bytes) 2024-05-06T00:28:33.682 INFO:teuthology.orchestra.run.smithi186.stdout:4119 punch from 0x88af400 to 0x88bb200, (0xbe00 bytes) 2024-05-06T00:28:33.683 INFO:teuthology.orchestra.run.smithi186.stdout:4121 read 0x56b3800 thru 0x56bf51e (0xbd1f bytes) 2024-05-06T00:28:33.683 INFO:teuthology.orchestra.run.smithi186.stdout:4122 write 0x84d5a00 thru 0x84da6b4 (0x4cb5 bytes) 2024-05-06T00:28:33.684 INFO:teuthology.orchestra.run.smithi186.stdout:4125 write 0x77a7400 thru 0x77ab0c2 (0x3cc3 bytes) 2024-05-06T00:28:33.696 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:33.697+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:33.711 INFO:teuthology.orchestra.run.smithi186.stdout:4129 trunc from 0xe572200 to 0x9fc00 2024-05-06T00:28:33.714 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:33.714 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:33.714 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:33.714 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:33.714 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:33.731 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:33.757 INFO:teuthology.orchestra.run.smithi186.stdout:4130 read 0x45800 thru 0x494fa (0x3cfb bytes) 2024-05-06T00:28:33.759 INFO:teuthology.orchestra.run.smithi186.stdout:4131 read 0x6c200 thru 0x732af (0x70b0 bytes) 2024-05-06T00:28:33.776 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:33.777+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:33.807 INFO:teuthology.orchestra.run.smithi186.stdout:4132 write 0xc99fa00 thru 0xc9abd98 (0xc399 bytes) 2024-05-06T00:28:33.808 INFO:teuthology.orchestra.run.smithi186.stdout:4134 read 0x4e40c00 thru 0x4e4b0b8 (0xa4b9 bytes) 2024-05-06T00:28:33.809 INFO:teuthology.orchestra.run.smithi186.stdout:4135 write 0x4c3f000 thru 0x4c4dd6c (0xed6d bytes) 2024-05-06T00:28:33.809 INFO:teuthology.orchestra.run.smithi186.stdout:4136 write 0x579ae00 thru 0x57a5465 (0xa666 bytes) 2024-05-06T00:28:33.810 INFO:teuthology.orchestra.run.smithi186.stdout:4138 read 0xbe86400 thru 0xbe9542e (0xf02f bytes) 2024-05-06T00:28:33.810 INFO:teuthology.orchestra.run.smithi186.stdout:4149 read 0x8588e00 thru 0x858af45 (0x2146 bytes) 2024-05-06T00:28:33.810 INFO:teuthology.orchestra.run.smithi186.stdout:4150 punch from 0x95dea00 to 0x95ea800, (0xbe00 bytes) 2024-05-06T00:28:33.812 INFO:teuthology.orchestra.run.smithi186.stdout:4153 write 0x68c0600 thru 0x68c8fb4 (0x89b5 bytes) 2024-05-06T00:28:33.812 INFO:teuthology.orchestra.run.smithi186.stdout:4154 write 0x7caae00 thru 0x7cafeec (0x50ed bytes) 2024-05-06T00:28:33.812 INFO:teuthology.orchestra.run.smithi186.stdout:4156 read 0xa162800 thru 0xa16e998 (0xc199 bytes) 2024-05-06T00:28:33.813 INFO:teuthology.orchestra.run.smithi186.stdout:4158 read 0x9116c00 thru 0x911dd13 (0x7114 bytes) 2024-05-06T00:28:33.813 INFO:teuthology.orchestra.run.smithi186.stdout:4161 read 0xf2d800 thru 0xf3aa68 (0xd269 bytes) 2024-05-06T00:28:33.813 INFO:teuthology.orchestra.run.smithi186.stdout:4163 read 0x8c9c600 thru 0x8ca9037 (0xca38 bytes) 2024-05-06T00:28:33.813 INFO:teuthology.orchestra.run.smithi186.stdout:4164 read 0x7307600 thru 0x7311c3e (0xa63f bytes) 2024-05-06T00:28:33.853 INFO:teuthology.orchestra.run.smithi186.stdout:4165 write 0xd679000 thru 0xd680ffb (0x7ffc bytes) 2024-05-06T00:28:33.855 INFO:teuthology.orchestra.run.smithi186.stdout:4166 write 0xb228000 thru 0xb234c47 (0xcc48 bytes) 2024-05-06T00:28:33.856 INFO:teuthology.orchestra.run.smithi186.stdout:4168 write 0x4ca2e00 thru 0x4ca82c0 (0x54c1 bytes) 2024-05-06T00:28:33.856 INFO:teuthology.orchestra.run.smithi186.stdout:4170 read 0x4c3a600 thru 0x4c3de8d (0x388e bytes) 2024-05-06T00:28:33.857 INFO:teuthology.orchestra.run.smithi186.stdout:4173 read 0xb4ef800 thru 0xb4f0dcd (0x15ce bytes) 2024-05-06T00:28:33.857 INFO:teuthology.orchestra.run.smithi186.stdout:4175 write 0x214a800 thru 0x215034b (0x5b4c bytes) 2024-05-06T00:28:33.857 INFO:teuthology.orchestra.run.smithi186.stdout:4177 write 0xc503c00 thru 0xc50acbb (0x70bc bytes) 2024-05-06T00:28:33.858 INFO:teuthology.orchestra.run.smithi186.stdout:4180 read 0x54a0c00 thru 0x54a625e (0x565f bytes) 2024-05-06T00:28:33.858 INFO:teuthology.orchestra.run.smithi186.stdout:4181 write 0x7e6a00 thru 0x7e798b (0xf8c bytes) 2024-05-06T00:28:33.860 INFO:teuthology.orchestra.run.smithi186.stdout:4182 read 0x7476e00 thru 0x747b838 (0x4a39 bytes) 2024-05-06T00:28:33.860 INFO:teuthology.orchestra.run.smithi186.stdout:4183 punch from 0x5adf000 to 0x5ae3e00, (0x4e00 bytes) 2024-05-06T00:28:33.861 INFO:teuthology.orchestra.run.smithi186.stdout:4186 read 0xb387200 thru 0xb39682c (0xf62d bytes) 2024-05-06T00:28:33.861 INFO:teuthology.orchestra.run.smithi186.stdout:4187 write 0x5922a00 thru 0x59275e4 (0x4be5 bytes) 2024-05-06T00:28:33.862 INFO:teuthology.orchestra.run.smithi186.stdout:4190 read 0x4c98e00 thru 0x4c98f68 (0x169 bytes) 2024-05-06T00:28:33.862 INFO:teuthology.orchestra.run.smithi186.stdout:4191 write 0x38ae600 thru 0x38b4492 (0x5e93 bytes) 2024-05-06T00:28:33.862 INFO:teuthology.orchestra.run.smithi186.stdout:4193 read 0x8085c00 thru 0x8092fa8 (0xd3a9 bytes) 2024-05-06T00:28:33.867 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:33.865+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:33.871 INFO:teuthology.orchestra.run.smithi186.stdout:4197 trunc from 0xd681000 to 0x8c19600 2024-05-06T00:28:33.903 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.896326608999999, 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:28:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:28:33.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.896098037000002, 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.908 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:28:33.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:28:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.896020753000002, 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:28:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:28:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.895824771999997, 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:28:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:28:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:28:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.988715396000003, 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:28:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:28:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.988654517000001, 2024-05-06T00:28:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:28:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.871098066999998, 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:28:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:28:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:28:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.871053338999999, 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:28:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:28:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.871014937999998, 2024-05-06T00:28:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:28:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:28:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.870977675999999, 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:28:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:28:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.870940767, 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.870902849, 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.870865733999999, 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:28:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.865085284999999, 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.865047200999999, 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.865009527000002, 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:28:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.855574016999999, 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:28:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.855541590000001, 2024-05-06T00:28:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:28:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.855517708000001, 2024-05-06T00:28:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:28:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:28:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.855497778, 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:28:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:28:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:28:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:33.989 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:33.990 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:33.992 INFO:teuthology.orchestra.run.smithi186.stdout:4199 read 0x8337800 thru 0x834659e (0xed9f bytes) 2024-05-06T00:28:33.992 INFO:teuthology.orchestra.run.smithi186.stdout:4202 write 0xc9d8400 thru 0xc9e2ee2 (0xaae3 bytes) 2024-05-06T00:28:33.992 INFO:teuthology.orchestra.run.smithi186.stdout:4204 write 0xc1dcc00 thru 0xc1dd187 (0x588 bytes) 2024-05-06T00:28:33.992 INFO:teuthology.orchestra.run.smithi186.stdout:4206 write 0x71e9600 thru 0x71ef060 (0x5a61 bytes) 2024-05-06T00:28:33.992 INFO:teuthology.orchestra.run.smithi186.stdout:4208 write 0x5f46c00 thru 0x5f48576 (0x1977 bytes) 2024-05-06T00:28:33.992 INFO:teuthology.orchestra.run.smithi186.stdout:4209 clone 26 order 23 su 32768 sc 14 2024-05-06T00:28:33.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:33.993 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:33.989+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:34.091 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:34.086+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:34.125 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:34.125 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:34.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:34.178 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:34.179 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:34.192 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:34.193+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:34.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:34.194 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:34.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:34.211 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:34.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:34.232 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:34.252 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:34.252 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:34.272 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:34.292 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:34.290+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:34.310 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:34.310 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:34.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:34.393 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:34.393+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:34.483 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:34.483 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:34.495 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:34.493+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:34.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:34.596 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:34.597+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:34.616 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:34.616 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:34.627 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:28:34.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:34.698 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:34.697+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:34.757 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:34.757 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:34.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:34.783 INFO:teuthology.orchestra.run.smithi186.stdout:leaving image image_client.0-clone25 intact 2024-05-06T00:28:34.817 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:34.814+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:34.892 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:34.893 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:34.900 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:34.894+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:34.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:35.001 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:35.001+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:35.075 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:35.075 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:35.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:35.137 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:35.134+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:35.203 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:35.201+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:35.210 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:35.211 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:35.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:35.313 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:35.310+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:35.395 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:35.395 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:35.405 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:35.405+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:35.409 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:35.506 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:35.502+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:35.547 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:35.547 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:35.563 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:35.608 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:35.602+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:35.609 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-05-06T00:28:35.684 INFO:teuthology.orchestra.run.smithi186.stdout:4210 read 0xa952000 thru 0xa95c48a (0xa48b bytes) 2024-05-06T00:28:35.684 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:35.684 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:35.685 INFO:teuthology.orchestra.run.smithi186.stdout:4211 write 0x5fb7a00 thru 0x5fc4815 (0xce16 bytes) 2024-05-06T00:28:35.687 INFO:teuthology.orchestra.run.smithi186.stdout:4212 write 0x9b5e00 thru 0x9b977f (0x3980 bytes) 2024-05-06T00:28:35.690 INFO:teuthology.orchestra.run.smithi186.stdout:4213 write 0xbfea200 thru 0xbff35ee (0x93ef bytes) 2024-05-06T00:28:35.693 INFO:teuthology.orchestra.run.smithi186.stdout:4215 read 0x6936e00 thru 0x693ec51 (0x7e52 bytes) 2024-05-06T00:28:35.694 INFO:teuthology.orchestra.run.smithi186.stdout:4217 read 0x7b58200 thru 0x7b5b662 (0x3463 bytes) 2024-05-06T00:28:35.699 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:35.744 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:35.745+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:35.759 INFO:teuthology.orchestra.run.smithi186.stdout:4220 trunc from 0xc9e3000 to 0x7735400 2024-05-06T00:28:35.810 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:35.806+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:35.834 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:35.834 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:35.851 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:35.874 INFO:teuthology.orchestra.run.smithi186.stdout:4221 read 0xcfba00 thru 0xcfcf5f (0x1560 bytes) 2024-05-06T00:28:35.911 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:35.906+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:35.942 INFO:teuthology.orchestra.run.smithi186.stdout:4223 trunc from 0x7735400 to 0x274c600 2024-05-06T00:28:36.013 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:36.013+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:36.035 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:36.035 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:36.049 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:36.061 INFO:teuthology.orchestra.run.smithi186.stdout:4225 write 0x5213a00 thru 0x5221c48 (0xe249 bytes) 2024-05-06T00:28:36.063 INFO:teuthology.orchestra.run.smithi186.stdout:4226 punch from 0x28cf400 to 0x28d5c00, (0x6800 bytes) 2024-05-06T00:28:36.072 INFO:teuthology.orchestra.run.smithi186.stdout:4227 trunc from 0x5221e00 to 0x90ec600 2024-05-06T00:28:36.100 INFO:teuthology.orchestra.run.smithi186.stdout:4229 trunc from 0x90ec600 to 0x5dae800 2024-05-06T00:28:36.124 INFO:teuthology.orchestra.run.smithi186.stdout:4231 write 0x43b5a00 thru 0x43c126c (0xb86d bytes) 2024-05-06T00:28:36.137 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:36.137+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:36.146 INFO:teuthology.orchestra.run.smithi186.stdout:4233 write 0x62d7c00 thru 0x62dbcf1 (0x40f2 bytes) 2024-05-06T00:28:36.149 INFO:teuthology.orchestra.run.smithi186.stdout:4235 write 0x5604800 thru 0x5611db7 (0xd5b8 bytes) 2024-05-06T00:28:36.151 INFO:teuthology.orchestra.run.smithi186.stdout:4236 punch from 0x558e000 to 0x5597200, (0x9200 bytes) 2024-05-06T00:28:36.173 INFO:teuthology.orchestra.run.smithi186.stdout:4237 write 0x6875200 thru 0x687a470 (0x5271 bytes) 2024-05-06T00:28:36.176 INFO:teuthology.orchestra.run.smithi186.stdout:4240 write 0x2cbee00 thru 0x2ccd078 (0xe279 bytes) 2024-05-06T00:28:36.178 INFO:teuthology.orchestra.run.smithi186.stdout:4242 write 0x5fece00 thru 0x5fee555 (0x1756 bytes) 2024-05-06T00:28:36.214 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:36.210+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:36.222 INFO:teuthology.orchestra.run.smithi186.stdout:4243 write 0xc591200 thru 0xc593ec7 (0x2cc8 bytes) 2024-05-06T00:28:36.225 INFO:teuthology.orchestra.run.smithi186.stdout:4252 write 0xee0e00 thru 0xeeed51 (0xdf52 bytes) 2024-05-06T00:28:36.226 INFO:teuthology.orchestra.run.smithi186.stdout:4255 punch from 0xbba0200 to 0xbbaf400, (0xf200 bytes) 2024-05-06T00:28:36.227 INFO:teuthology.orchestra.run.smithi186.stdout:4257 punch from 0x61be000 to 0x61bf000, (0x1000 bytes) 2024-05-06T00:28:36.227 INFO:teuthology.orchestra.run.smithi186.stdout:4259 write 0x891c800 thru 0x8925721 (0x8f22 bytes) 2024-05-06T00:28:36.228 INFO:teuthology.orchestra.run.smithi186.stdout:4261 write 0x56a1800 thru 0x56ae2c0 (0xcac1 bytes) 2024-05-06T00:28:36.229 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:36.229 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:36.232 INFO:teuthology.orchestra.run.smithi186.stdout:4262 read 0x3502e00 thru 0x350338d (0x58e bytes) 2024-05-06T00:28:36.233 INFO:teuthology.orchestra.run.smithi186.stdout:4264 read 0x59ade00 thru 0x59b9530 (0xb731 bytes) 2024-05-06T00:28:36.234 INFO:teuthology.orchestra.run.smithi186.stdout:4266 read 0x25a600 thru 0x25c86a (0x226b bytes) 2024-05-06T00:28:36.239 INFO:teuthology.orchestra.run.smithi186.stdout:4268 read 0x8c2ca00 thru 0x8c2dd18 (0x1319 bytes) 2024-05-06T00:28:36.239 INFO:teuthology.orchestra.run.smithi186.stdout:4269 punch from 0x5d21200 to 0x5d28e00, (0x7c00 bytes) 2024-05-06T00:28:36.245 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:36.249 INFO:teuthology.orchestra.run.smithi186.stdout:4270 trunc from 0xc594000 to 0x7b34600 2024-05-06T00:28:36.269 INFO:teuthology.orchestra.run.smithi186.stdout:4271 write 0x472ee00 thru 0x4735486 (0x6687 bytes) 2024-05-06T00:28:36.274 INFO:teuthology.orchestra.run.smithi186.stdout:4272 punch from 0x5e57a00 to 0x5e5d400, (0x5a00 bytes) 2024-05-06T00:28:36.290 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:36.297 INFO:teuthology.orchestra.run.smithi186.stdout:4273 write 0xa91a000 thru 0xa928457 (0xe458 bytes) 2024-05-06T00:28:36.298 INFO:teuthology.orchestra.run.smithi186.stdout:4274 read 0x5e06800 thru 0x5e11488 (0xac89 bytes) 2024-05-06T00:28:36.340 INFO:teuthology.orchestra.run.smithi186.stdout:4276 write 0xd07fe00 thru 0xd08bb26 (0xbd27 bytes) 2024-05-06T00:28:36.341 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:36.341+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:36.344 INFO:teuthology.orchestra.run.smithi186.stdout:4281 read 0x1123200 thru 0x11302d6 (0xd0d7 bytes) 2024-05-06T00:28:36.345 INFO:teuthology.orchestra.run.smithi186.stdout:4283 punch from 0xa8e1400 to 0xa8ea400, (0x9000 bytes) 2024-05-06T00:28:36.385 INFO:teuthology.orchestra.run.smithi186.stdout:4285 write 0xea15400 thru 0xea185ef (0x31f0 bytes) 2024-05-06T00:28:36.397 INFO:teuthology.orchestra.run.smithi186.stdout:4286 trunc from 0xea18600 to 0xb2eee00 2024-05-06T00:28:36.416 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:36.413+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:36.423 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:36.423 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:36.439 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:36.440 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:36.440 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:36.442 INFO:teuthology.orchestra.run.smithi186.stdout:4289 read 0x528de00 thru 0x5297112 (0x9313 bytes) 2024-05-06T00:28:36.446 INFO:teuthology.orchestra.run.smithi186.stdout:4290 write 0x95400 thru 0x9e86e (0x946f bytes) 2024-05-06T00:28:36.448 INFO:teuthology.orchestra.run.smithi186.stdout:4292 read 0x7a9d000 thru 0x7a9e130 (0x1131 bytes) 2024-05-06T00:28:36.448 INFO:teuthology.orchestra.run.smithi186.stdout:4294 punch from 0x3dd7a00 to 0x3de3800, (0xbe00 bytes) 2024-05-06T00:28:36.449 INFO:teuthology.orchestra.run.smithi186.stdout:4296 write 0x5bb8000 thru 0x5bbd6c2 (0x56c3 bytes) 2024-05-06T00:28:36.456 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:36.457 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:36.460 INFO:teuthology.orchestra.run.smithi186.stdout:4299 trunc from 0xb2eee00 to 0x5f5a800 2024-05-06T00:28:36.477 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:36.477 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:36.500 INFO:teuthology.orchestra.run.smithi186.stdout:4300 read 0x1557800 thru 0x1559c3b (0x243c bytes) 2024-05-06T00:28:36.503 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:36.513 INFO:teuthology.orchestra.run.smithi186.stdout:4301 trunc from 0x5f5a800 to 0xa8bda00 2024-05-06T00:28:36.516 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:36.510+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:36.525 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:36.527 INFO:teuthology.orchestra.run.smithi186.stdout:4303 punch from 0x7f93600 to 0x7fa3400, (0xfe00 bytes) 2024-05-06T00:28:36.529 INFO:teuthology.orchestra.run.smithi186.stdout:4305 punch from 0x65d5e00 to 0x65d8000, (0x2200 bytes) 2024-05-06T00:28:36.529 INFO:teuthology.orchestra.run.smithi186.stdout:4307 read 0xa02c00 thru 0xa082a0 (0x56a1 bytes) 2024-05-06T00:28:36.536 INFO:teuthology.orchestra.run.smithi186.stdout:4308 trunc from 0xa8bda00 to 0xcbf6800 2024-05-06T00:28:36.570 INFO:teuthology.orchestra.run.smithi186.stdout:4309 trunc from 0xcbf6800 to 0x7350200 2024-05-06T00:28:36.583 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:36.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:36.583 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:36.583 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:36.597 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:36.621 INFO:teuthology.orchestra.run.smithi186.stdout:4310 write 0xb99e400 thru 0xb9a6ae2 (0x86e3 bytes) 2024-05-06T00:28:36.622 INFO:teuthology.orchestra.run.smithi186.stdout:4311 read 0x1d8c00 thru 0x1dd8e6 (0x4ce7 bytes) 2024-05-06T00:28:36.640 INFO:teuthology.orchestra.run.smithi186.stdout:4312 trunc from 0xb9a6c00 to 0x4ba5400 2024-05-06T00:28:36.641 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:36.641+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:36.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:28:36.706 INFO:teuthology.orchestra.run.smithi186.stdout:4314 write 0x71c8c00 thru 0x71d51d0 (0xc5d1 bytes) 2024-05-06T00:28:36.709 INFO:teuthology.orchestra.run.smithi186.stdout:4315 read 0x34fea00 thru 0x3503183 (0x4784 bytes) 2024-05-06T00:28:36.718 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:36.714+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:36.725 INFO:teuthology.orchestra.run.smithi186.stdout:4316 trunc from 0x71d5200 to 0xcb8ec00 2024-05-06T00:28:36.749 INFO:teuthology.orchestra.run.smithi186.stdout:4317 read 0x15f4c00 thru 0x1602e91 (0xe292 bytes) 2024-05-06T00:28:36.752 INFO:teuthology.orchestra.run.smithi186.stdout:4319 read 0x6e51400 thru 0x6e591c5 (0x7dc6 bytes) 2024-05-06T00:28:36.759 INFO:teuthology.orchestra.run.smithi186.stdout:4321 trunc from 0xcb8ec00 to 0xa2db800 2024-05-06T00:28:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:36.780 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:36.780 INFO:teuthology.orchestra.run.smithi186.stdout:4322 write 0x1d1e200 thru 0x1d2873d (0xa53e bytes) 2024-05-06T00:28:36.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:36.806 INFO:teuthology.orchestra.run.smithi186.stdout:4323 write 0xae6d000 thru 0xae75fcb (0x8fcc bytes) 2024-05-06T00:28:36.807 INFO:teuthology.orchestra.run.smithi186.stdout:4324 read 0x382ac00 thru 0x382ac13 (0x14 bytes) 2024-05-06T00:28:36.808 INFO:teuthology.orchestra.run.smithi186.stdout:4326 read 0x8a4a800 thru 0x8a56626 (0xbe27 bytes) 2024-05-06T00:28:36.809 INFO:teuthology.orchestra.run.smithi186.stdout:4332 punch from 0x7d4c000 to 0x7d4f200, (0x3200 bytes) 2024-05-06T00:28:36.809 INFO:teuthology.orchestra.run.smithi186.stdout:4334 read 0x6a8b600 thru 0x6a92979 (0x737a bytes) 2024-05-06T00:28:36.811 INFO:teuthology.orchestra.run.smithi186.stdout:4336 read 0x5e11c00 thru 0x5e21003 (0xf404 bytes) 2024-05-06T00:28:36.812 INFO:teuthology.orchestra.run.smithi186.stdout:4338 punch from 0x9d68c00 to 0x9d76000, (0xd400 bytes) 2024-05-06T00:28:36.840 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:36.841+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:36.842 INFO:teuthology.orchestra.run.smithi186.stdout:4339 write 0xe333000 thru 0xe335c5f (0x2c60 bytes) 2024-05-06T00:28:36.843 INFO:teuthology.orchestra.run.smithi186.stdout:4341 punch from 0xb8c3200 to 0xb8cdc00, (0xaa00 bytes) 2024-05-06T00:28:36.843 INFO:teuthology.orchestra.run.smithi186.stdout:4343 read 0x80bb400 thru 0x80c8775 (0xd376 bytes) 2024-05-06T00:28:36.847 INFO:teuthology.orchestra.run.smithi186.stdout:4344 read 0x57d9800 thru 0x57e80af (0xe8b0 bytes) 2024-05-06T00:28:36.849 INFO:teuthology.orchestra.run.smithi186.stdout:4347 read 0x2112200 thru 0x2116242 (0x4043 bytes) 2024-05-06T00:28:36.850 INFO:teuthology.orchestra.run.smithi186.stdout:4349 punch from 0xaf6ac00 to 0xaf6fa00, (0x4e00 bytes) 2024-05-06T00:28:36.850 INFO:teuthology.orchestra.run.smithi186.stdout:4352 read 0xa510800 thru 0xa514f2a (0x472b bytes) 2024-05-06T00:28:36.853 INFO:teuthology.orchestra.run.smithi186.stdout:4362 punch from 0xbcd6800 to 0xbce0e00, (0xa600 bytes) 2024-05-06T00:28:36.854 INFO:teuthology.orchestra.run.smithi186.stdout:4363 read 0x145e600 thru 0x146b891 (0xd292 bytes) 2024-05-06T00:28:36.856 INFO:teuthology.orchestra.run.smithi186.stdout:4364 write 0x79ee200 thru 0x79fb97e (0xd77f bytes) 2024-05-06T00:28:36.859 INFO:teuthology.orchestra.run.smithi186.stdout:4368 write 0x8af800 thru 0x8b3fb2 (0x47b3 bytes) 2024-05-06T00:28:36.861 INFO:teuthology.orchestra.run.smithi186.stdout:4369 punch from 0x84b6c00 to 0x84b7c00, (0x1000 bytes) 2024-05-06T00:28:36.861 INFO:teuthology.orchestra.run.smithi186.stdout:4370 read 0xc442c00 thru 0xc443352 (0x753 bytes) 2024-05-06T00:28:36.862 INFO:teuthology.orchestra.run.smithi186.stdout:4372 clone 27 order 25 su 1048576 sc 11 2024-05-06T00:28:36.921 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:36.921+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:36.962 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.954810881, 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:36.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:28:36.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.967 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.954582309000003, 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:36.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:28:36.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:28:36.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:28:36.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:36.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:28:36.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:28:36.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.972 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:36.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.954505025000003, 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:28:36.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:36.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.954309043999999, 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:36.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:36.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.047199667999998, 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:36.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:28:36.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:28:36.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.047138789000002, 2024-05-06T00:28:36.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:28:36.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:36.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:36.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.929582339, 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:28:36.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:36.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:28:36.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.929537611000001, 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:28:36.995 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:28:36.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:36.999 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.929499210000003, 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:28:37.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.929461947999997, 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:28:37.004 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:28:37.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.929425039000002, 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:28:37.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:28:37.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.929387120999998, 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.013 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:28:37.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:28:37.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.929350006, 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:28:37.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:28:37.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.923569557, 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.026 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.923531472999997, 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:37.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:28:37.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.923493798999999, 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:37.031 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:37.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.914058289000003, 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:37.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.914025862000003, 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:28:37.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:28:37.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.914001980000002, 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:28:37.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:28:37.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.913982050000001, 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:37.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:28:37.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:37.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:37.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:28:37.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:28:37.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.052 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:37.052 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:37.052 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:37.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:37.054 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:37.054+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:37.155 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:37.150+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:37.184 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:37.185 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:37.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:37.285 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:37.282+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:37.351 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 0, 5, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:37.352 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:37.352 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-05-06T00:28:37.352 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-06T00:28:37.352 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-06T00:28:37.352 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-05-06T00:28:37.365 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:37.365 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:37.366 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:37.381 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:37.400 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:37.401+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:37.469 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:37.466+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:37.485 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-06T00:28:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:37.550 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 127.94729526899999, 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:28:37.551 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:28:37.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:28:37.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.674646569999993, 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:28:37.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:28:37.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.674611393999996, 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.564 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:28:37.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:28:37.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.674522714999995, 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:28:37.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:28:37.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.674420772000005, 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:28:37.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:28:37.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:28:37.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:28:37.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:28:37.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.674311669999994, 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.582 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:28:37.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.674213811000001, 2024-05-06T00:28:37.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.674148760999998, 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:28:37.594 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:37.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:28:37.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.674056114999999, 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.673984110999996, 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:28:37.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:28:37.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.612 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914522625000004, 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:37.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:28:37.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914503377000003, 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.619 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:28:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:28:37.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914453257000005, 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:28:37.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:28:37.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914432206000001, 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.631 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:37.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.636 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914416887000002, 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:28:37.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:28:37.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:28:37.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914400005000005, 2024-05-06T00:28:37.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:37.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:28:37.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:28:37.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914381668999994, 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.648 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914365666999998, 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:28:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:28:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:28:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.914279304000004, 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.660 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:28:37.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:28:37.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.904316828000006, 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.666 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:28:37.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:37.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:28:37.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:28:37.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.672 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.672 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:28:37.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:37.675 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:37.673+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:37.774 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:37.770+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:37.811 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone26 from 0xe335e00 (overlap 0x274c600) to 0x13a6200 2024-05-06T00:28:37.860 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:37.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:37.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.860 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5356.0:2813 3.11 3:88ff6aec:::rbd_object_map.110c69635521:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-05-06T00:28:37.860 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:37.858525+0000", 2024-05-06T00:28:37.860 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0019761729999999999, 2024-05-06T00:28:37.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002044255, 2024-05-06T00:28:37.860 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5356", 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1796886038", 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2813 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.858525+0000", 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.858525+0000", 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.858526+0000", 2024-05-06T00:28:37.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3289999999999999e-06 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.858541+0000", 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4216000000000001e-05 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.858554+0000", 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3029e-05 2024-05-06T00:28:37.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.858562+0000", 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7399999999999993e-06 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.858635+0000", 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2972999999999994e-05 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.858777+0000", 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014131199999999999 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859381+0000", 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00060450499999999997 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.866 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5356.0:2814 3.11 3:88ff6aec:::rbd_object_map.110c69635521:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-05-06T00:28:37.866 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:37.859152+0000", 2024-05-06T00:28:37.866 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.001348737, 2024-05-06T00:28:37.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016708770000000001, 2024-05-06T00:28:37.866 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.866 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5356", 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1796886038", 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2814 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859152+0000", 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859152+0000", 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859155+0000", 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1100000000000001e-06 2024-05-06T00:28:37.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859165+0000", 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346000000000001e-05 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859187+0000", 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2517000000000001e-05 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859201+0000", 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3540000000000001e-05 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.860313+0000", 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001112329 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:37.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.860510+0000", 2024-05-06T00:28:37.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000196627 2024-05-06T00:28:37.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.871 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5356.0:2815 3.11 3.3756ff11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:37.859316+0000", 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0011849790000000001, 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617348, 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "queued for pg", 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5356", 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/1796886038", 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2815 2024-05-06T00:28:37.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859316+0000", 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859316+0000", 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859318+0000", 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.327e-06 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859332+0000", 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3978999999999999e-05 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859346+0000", 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4304000000000001e-05 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.859355+0000", 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3300000000000005e-06 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop_reply(client.5356.0:2813 3.11 e110/105)", 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:37.860893+0000", 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4294967295.9996076, 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000121623, 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:37.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "queued for pg", 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.860893+0000", 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.860893+0000", 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.860894+0000", 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9999999999999997e-07 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.860898+0000", 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9060000000000004e-06 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.860913+0000", 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.506e-05 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:37.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:37.860919+0000", 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4360000000000001e-06 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-05-06T00:28:37.878 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 4 2024-05-06T00:28:37.879 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:37.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:37.881 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:37.877+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:37.981 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:37.978+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:38.046 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:38.046 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:38.046 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:38.047 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:38.047 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:38.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:38.082 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:38.081+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:38.183 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:38.181+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:38.235 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:38.235 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.573650647999997, 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:28:38.236 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:28:38.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:28:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.594409143, 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:28:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:28:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:28:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:28:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.594334924000002, 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:28:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.594273025, 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.594213426000003, 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:28:38.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:38.261 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.594129291000002, 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.594063777999999, 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.266 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:28:38.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.270 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.594005916, 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:38.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593909320999998, 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:28:38.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:28:38.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593831596000001, 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:28:38.280 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:28:38.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.285 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593758809999997, 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:38.291 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593681318999998, 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.292 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593587190000001, 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:28:38.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:28:38.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:28:38.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:28:38.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:28:38.302 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.302 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.302 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593504228, 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.303 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.304 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:28:38.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.304 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:28:38.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593430122999997, 2024-05-06T00:28:38.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:28:38.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:28:38.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:28:38.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593330934000001, 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.313 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:38.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.317 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.318 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.318 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.318 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593267781999998, 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:28:38.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:28:38.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:28:38.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:28:38.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593167516999998, 2024-05-06T00:28:38.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.325 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:28:38.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.593078145, 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.330 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.347219461999998, 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:28:38.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:28:38.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.339 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:38.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:38.341 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:38.338+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:38.442 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:38.438+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:38.515 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:38.516 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:38.516 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:38.516 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:38.529 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:38.534 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-05-06T00:28:38.543 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:38.541+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:38.647 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:38.645+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:38.705 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:38.705 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:38.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:38.745 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:38.741+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:38.793 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:38.844 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:38.838+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:38.862 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.401783859999995, 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.863 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:28:38.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:28:38.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.400981870999999, 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.868 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.872 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.400895417000001, 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.873 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:28:38.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.400822824000002, 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:28:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:28:38.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:28:38.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.400762972999999, 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.882 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:28:38.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.400687289999993, 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:28:38.886 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.890 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.392981716999998, 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:28:38.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:28:38.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.392927510000007, 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:28:38.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.899 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.392875360999994, 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:28:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.390917646000005, 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:28:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.390841305999999, 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:28:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.575703598999993, 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:38.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:28:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:28:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:28:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:28:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:28:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:28:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:38.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:28:38.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:28:38.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:28:38.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.575651157999999, 2024-05-06T00:28:38.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:28:38.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:28:38.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:28:38.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.928 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.929 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.930 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:38.930 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:38.931 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.568631174999993, 2024-05-06T00:28:38.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:28:38.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.937 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.568584463999997, 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:28:38.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:38.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.568254355999997, 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.944 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:28:38.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:28:38.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.568226433999996, 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.950 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:28:38.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.557359544999997, 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.955 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.899709744000001, 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.959 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:28:38.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:38.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.964 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.898525128999999, 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:28:38.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:28:38.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:38.969 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:38.970 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:38.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:38.971 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:38.972 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:38.972 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:38.972 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:38.966+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:38.973 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:38.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:38.989 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:39.011 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:39.011 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:39.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:39.058 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:39.058 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:39.072 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:39.066+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:39.103 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:39.103 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:39.118 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:39.173 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:39.170+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:39.274 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:39.270+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:39.278 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:39.278 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:39.292 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:39.375 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:39.373+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:39.421 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:39.421 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:39.437 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:39.480 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:39.481+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:39.578 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:39.574+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:39.613 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:39.613 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:39.628 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:39.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:28:39.678 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:39.674+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:39.753 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:39.753 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:39.769 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:39.812 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:39.813+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:39.880 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:39.874+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:39.883 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:39.883 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:39.897 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:39.973 INFO:teuthology.orchestra.run.smithi186.stdout:4374 write 0x983a00 thru 0x9923f7 (0xe9f8 bytes) 2024-05-06T00:28:39.974 INFO:teuthology.orchestra.run.smithi186.stdout:4376 read 0x3107e00 thru 0x310c6db (0x48dc bytes) 2024-05-06T00:28:39.976 INFO:teuthology.orchestra.run.smithi186.stdout:4379 write 0xe0df400 thru 0xe0e020a (0xe0b bytes) 2024-05-06T00:28:39.977 INFO:teuthology.orchestra.run.smithi186.stdout:4380 read 0x4709000 thru 0x4712705 (0x9706 bytes) 2024-05-06T00:28:39.979 INFO:teuthology.orchestra.run.smithi186.stdout:4381 write 0xb4ca600 thru 0xb4d18ab (0x72ac bytes) 2024-05-06T00:28:39.981 INFO:teuthology.orchestra.run.smithi186.stdout:4382 write 0x8a2ac00 thru 0x8a38b7e (0xdf7f bytes) 2024-05-06T00:28:39.982 INFO:teuthology.orchestra.run.smithi186.stdout:4384 write 0xc355a00 thru 0xc35a3d5 (0x49d6 bytes) 2024-05-06T00:28:39.983 INFO:teuthology.orchestra.run.smithi186.stdout:4387 write 0x2887000 thru 0x288f09d (0x809e bytes) 2024-05-06T00:28:39.985 INFO:teuthology.orchestra.run.smithi186.stdout:4388 write 0xcc7e200 thru 0xcc80b56 (0x2957 bytes) 2024-05-06T00:28:39.986 INFO:teuthology.orchestra.run.smithi186.stdout:4389 write 0x4964400 thru 0x496b94b (0x754c bytes) 2024-05-06T00:28:39.988 INFO:teuthology.orchestra.run.smithi186.stdout:4391 punch from 0x4b66e00 to 0x4b73e00, (0xd000 bytes) 2024-05-06T00:28:39.988 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:39.989+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:40.081 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:40.081+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:40.082 INFO:teuthology.orchestra.run.smithi186.stdout:4394 read 0xd866c00 thru 0xd86c469 (0x586a bytes) 2024-05-06T00:28:40.083 INFO:teuthology.orchestra.run.smithi186.stdout:4395 write 0x16d4800 thru 0x16dc692 (0x7e93 bytes) 2024-05-06T00:28:40.084 INFO:teuthology.orchestra.run.smithi186.stdout:4397 write 0x3f58600 thru 0x3f65d1d (0xd71e bytes) 2024-05-06T00:28:40.085 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:40.085 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:40.087 INFO:teuthology.orchestra.run.smithi186.stdout:4398 trunc from 0xe335e00 to 0xe010a00 2024-05-06T00:28:40.103 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:40.149 INFO:teuthology.orchestra.run.smithi186.stdout:4399 write 0x76bd800 thru 0x76beab6 (0x12b7 bytes) 2024-05-06T00:28:40.150 INFO:teuthology.orchestra.run.smithi186.stdout:4400 read 0x5eeb600 thru 0x5ef9fca (0xe9cb bytes) 2024-05-06T00:28:40.152 INFO:teuthology.orchestra.run.smithi186.stdout:4402 write 0x9588a00 thru 0x958a86e (0x1e6f bytes) 2024-05-06T00:28:40.152 INFO:teuthology.orchestra.run.smithi186.stdout:4405 read 0x2282400 thru 0x22850f9 (0x2cfa bytes) 2024-05-06T00:28:40.153 INFO:teuthology.orchestra.run.smithi186.stdout:4407 write 0xde30e00 thru 0xde3d47c (0xc67d bytes) 2024-05-06T00:28:40.156 INFO:teuthology.orchestra.run.smithi186.stdout:4408 read 0xb28de00 thru 0xb29a6fd (0xc8fe bytes) 2024-05-06T00:28:40.157 INFO:teuthology.orchestra.run.smithi186.stdout:4410 punch from 0xab27200 to 0xab35400, (0xe200 bytes) 2024-05-06T00:28:40.165 INFO:teuthology.orchestra.run.smithi186.stdout:4412 punch from 0xd23ce00 to 0xd241400, (0x4600 bytes) 2024-05-06T00:28:40.171 INFO:teuthology.orchestra.run.smithi186.stdout:4413 write 0xda39600 thru 0xda44ccc (0xb6cd bytes) 2024-05-06T00:28:40.175 INFO:teuthology.orchestra.run.smithi186.stdout:4414 write 0xc8ce400 thru 0xc8d73d4 (0x8fd5 bytes) 2024-05-06T00:28:40.175 INFO:teuthology.orchestra.run.smithi186.stdout:4415 punch from 0x8764800 to 0x8765e00, (0x1600 bytes) 2024-05-06T00:28:40.182 INFO:teuthology.orchestra.run.smithi186.stdout:4416 read 0xab63e00 thru 0xab64e0b (0x100c bytes) 2024-05-06T00:28:40.186 INFO:teuthology.orchestra.run.smithi186.stdout:4418 write 0x1f36e00 thru 0x1f42d99 (0xbf9a bytes) 2024-05-06T00:28:40.192 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:40.193+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:40.223 INFO:teuthology.orchestra.run.smithi186.stdout:4419 trunc from 0xe010a00 to 0x8154800 2024-05-06T00:28:40.277 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:40.277 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:40.279 INFO:teuthology.orchestra.run.smithi186.stdout:4421 trunc from 0x8154800 to 0x9dcfe00 2024-05-06T00:28:40.283 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:40.281+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:40.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:40.303 INFO:teuthology.orchestra.run.smithi186.stdout:4422 punch from 0xd2ca00 to 0xd30a00, (0x4000 bytes) 2024-05-06T00:28:40.304 INFO:teuthology.orchestra.run.smithi186.stdout:4424 read 0x4bc5e00 thru 0x4bc7c3c (0x1e3d bytes) 2024-05-06T00:28:40.304 INFO:teuthology.orchestra.run.smithi186.stdout:4425 read 0x53dec00 thru 0x53e396d (0x4d6e bytes) 2024-05-06T00:28:40.306 INFO:teuthology.orchestra.run.smithi186.stdout:4427 read 0x59f3c00 thru 0x59fab9e (0x6f9f bytes) 2024-05-06T00:28:40.306 INFO:teuthology.orchestra.run.smithi186.stdout:4433 write 0x1655e00 thru 0x165a60e (0x480f bytes) 2024-05-06T00:28:40.307 INFO:teuthology.orchestra.run.smithi186.stdout:4436 punch from 0x819200 to 0x823600, (0xa400 bytes) 2024-05-06T00:28:40.334 INFO:teuthology.orchestra.run.smithi186.stdout:4438 write 0xe73f400 thru 0xe747ef5 (0x8af6 bytes) 2024-05-06T00:28:40.339 INFO:teuthology.orchestra.run.smithi186.stdout:4440 read 0xccd8a00 thru 0xcce2bef (0xa1f0 bytes) 2024-05-06T00:28:40.340 INFO:teuthology.orchestra.run.smithi186.stdout:4443 punch from 0xd2dcc00 to 0xd2df800, (0x2c00 bytes) 2024-05-06T00:28:40.341 INFO:teuthology.orchestra.run.smithi186.stdout:4444 punch from 0x2ce2a00 to 0x2ceda00, (0xb000 bytes) 2024-05-06T00:28:40.341 INFO:teuthology.orchestra.run.smithi186.stdout:4447 read 0x638c000 thru 0x639180b (0x580c bytes) 2024-05-06T00:28:40.345 INFO:teuthology.orchestra.run.smithi186.stdout:4448 read 0x2aafa00 thru 0x2abac41 (0xb242 bytes) 2024-05-06T00:28:40.346 INFO:teuthology.orchestra.run.smithi186.stdout:4449 write 0x87d1c00 thru 0x87d916a (0x756b bytes) 2024-05-06T00:28:40.347 INFO:teuthology.orchestra.run.smithi186.stdout:4450 write 0x506e600 thru 0x50745b3 (0x5fb4 bytes) 2024-05-06T00:28:40.348 INFO:teuthology.orchestra.run.smithi186.stdout:4452 read 0x4337200 thru 0x4346a94 (0xf895 bytes) 2024-05-06T00:28:40.383 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:40.381+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:40.384 INFO:teuthology.orchestra.run.smithi186.stdout:4453 trunc from 0xe748000 to 0x3f5aa00 2024-05-06T00:28:40.461 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:40.461 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:40.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:40.484 INFO:teuthology.orchestra.run.smithi186.stdout:4454 write 0x9224400 thru 0x92330e4 (0xece5 bytes) 2024-05-06T00:28:40.492 INFO:teuthology.orchestra.run.smithi186.stdout:4455 trunc from 0x9233200 to 0x6e07a00 2024-05-06T00:28:40.520 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:40.521+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:40.522 INFO:teuthology.orchestra.run.smithi186.stdout:4457 punch from 0x48cec00 to 0x48dd000, (0xe400 bytes) 2024-05-06T00:28:40.523 INFO:teuthology.orchestra.run.smithi186.stdout:4458 punch from 0x318c200 to 0x319ae00, (0xec00 bytes) 2024-05-06T00:28:40.523 INFO:teuthology.orchestra.run.smithi186.stdout:4459 read 0xbc4600 thru 0xbcc16c (0x7b6d bytes) 2024-05-06T00:28:40.526 INFO:teuthology.orchestra.run.smithi186.stdout:4465 write 0x57f1600 thru 0x57fd8b1 (0xc2b2 bytes) 2024-05-06T00:28:40.527 INFO:teuthology.orchestra.run.smithi186.stdout:4468 clone 28 order 21 su 131072 sc 5 2024-05-06T00:28:40.586 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:40.585+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:40.613 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:40.613 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:40.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:40.665 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:28:40.688 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:40.685+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:40.753 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:40.753 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:40.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:40.788 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:40.789+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:40.859 INFO:teuthology.orchestra.run.smithi186.stdout:flattening image image_client.0-clone27 2024-05-06T00:28:40.889 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:40.889+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:40.895 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:40.895 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:40.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:40.990 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:40.989+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:41.032 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:41.033 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:41.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:41.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:41.125 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:41.125+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:41.170 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:41.170 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:41.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:41.191 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:41.189+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:41.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:41.233 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:41.251 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:41.267 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:41.268 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:41.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:41.289 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:41.292 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:41.293+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:41.304 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:41.304 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:41.311 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:41.320 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:41.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:41.334 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:41.396 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:41.397+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:41.446 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:41.446 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:41.446 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:41.446 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:41.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:41.504 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-05-06T00:28:41.505 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:41.502+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:41.595 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:41.593+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:41.596 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:41.596 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:41.597 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:41.597 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:41.597 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:41.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:41.696 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:41.697+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:41.788 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.326850602999997, 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:28:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:28:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:28:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.326048614000001, 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:28:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:28:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.325962160000003, 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:28:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:28:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.325889567000004, 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:28:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:28:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.325829716000001, 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.325754032999996, 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:28:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.31804846, 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.317994252999995, 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:28:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:28:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.317942103999997, 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:28:41.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:28:41.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.315984388999993, 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.828 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.315908049000001, 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:28:41.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.500770341999996, 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:28:41.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.500717901000002, 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:28:41.842 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:41.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:28:41.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:28:41.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.493697917999995, 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:28:41.848 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:28:41.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:41.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:28:41.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:41.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:41.854 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.493651206999999, 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.493321098999999, 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:28:41.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.865 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.493293176999998, 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:41.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:28:41.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:28:41.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:41.870 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.482426287999999, 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.824776487000001, 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:28:41.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:28:41.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:28:41.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.823591872000002, 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:28:41.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:28:41.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:28:41.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:28:41.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.885 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:41.885 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:41.885 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:41.886 INFO:teuthology.orchestra.run.smithi186.stdout:4471 trunc from 0x6e07a00 to 0xae5fc00 2024-05-06T00:28:41.886 INFO:teuthology.orchestra.run.smithi186.stdout:4472 trunc from 0xae5fc00 to 0x4a5cc00 2024-05-06T00:28:41.887 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:41.882+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:41.887 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:41.904 INFO:teuthology.orchestra.run.smithi186.stdout:4473 punch from 0x3425400 to 0x3433600, (0xe200 bytes) 2024-05-06T00:28:41.907 INFO:teuthology.orchestra.run.smithi186.stdout:4474 read 0xfec800 thru 0xfedbc0 (0x13c1 bytes) 2024-05-06T00:28:41.928 INFO:teuthology.orchestra.run.smithi186.stdout:4475 trunc from 0x4a5cc00 to 0x3058600 2024-05-06T00:28:41.987 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:41.985+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:42.030 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:42.030 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:42.030 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:42.030 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:42.031 INFO:teuthology.orchestra.run.smithi186.stdout:4476 write 0xb58e800 thru 0xb591a71 (0x3272 bytes) 2024-05-06T00:28:42.044 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:42.046 INFO:teuthology.orchestra.run.smithi186.stdout:4477 trunc from 0xb591c00 to 0xc438e00 2024-05-06T00:28:42.070 INFO:teuthology.orchestra.run.smithi186.stdout:4478 punch from 0x6f9b200 to 0x6fa4a00, (0x9800 bytes) 2024-05-06T00:28:42.072 INFO:teuthology.orchestra.run.smithi186.stdout:4482 write 0x8b8a200 thru 0x8b97e97 (0xdc98 bytes) 2024-05-06T00:28:42.072 INFO:teuthology.orchestra.run.smithi186.stdout:4483 read 0x14ba400 thru 0x14bba6a (0x166b bytes) 2024-05-06T00:28:42.073 INFO:teuthology.orchestra.run.smithi186.stdout:4486 write 0xa80e400 thru 0xa81206f (0x3c70 bytes) 2024-05-06T00:28:42.073 INFO:teuthology.orchestra.run.smithi186.stdout:4489 write 0x7708600 thru 0x770a104 (0x1b05 bytes) 2024-05-06T00:28:42.088 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:42.086+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:42.110 INFO:teuthology.orchestra.run.smithi186.stdout:4491 trunc from 0xc438e00 to 0x1d4000 2024-05-06T00:28:42.190 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:42.186+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:42.203 INFO:teuthology.orchestra.run.smithi186.stdout:4492 write 0x6389200 thru 0x6398fd9 (0xfdda bytes) 2024-05-06T00:28:42.205 INFO:teuthology.orchestra.run.smithi186.stdout:4494 read 0x128e400 thru 0x128fc0c (0x180d bytes) 2024-05-06T00:28:42.206 INFO:teuthology.orchestra.run.smithi186.stdout:4495 write 0x200cc00 thru 0x201a7d5 (0xdbd6 bytes) 2024-05-06T00:28:42.206 INFO:teuthology.orchestra.run.smithi186.stdout:4496 punch from 0x6c3200 to 0x6c6c00, (0x3a00 bytes) 2024-05-06T00:28:42.207 INFO:teuthology.orchestra.run.smithi186.stdout:4499 clone 29 order 25 su 262144 sc 13 2024-05-06T00:28:42.217 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:42.218 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:42.235 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:42.291 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:42.286+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.55326648400001, 2024-05-06T00:28:42.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:28:42.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:28:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.553250281, 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:28:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:28:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:28:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34113 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 845824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.545120199000003, 2024-05-06T00:28:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.073299384, 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34113 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2200000000000004e-07 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812509+0000", 2024-05-06T00:28:42.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-06 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812519+0000", 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6979999999999993e-06 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812528+0000", 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5370000000000003e-06 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817889+0000", 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053609809999999999 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817903+0000", 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3545e-05 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885659+0000", 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.067755797 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885734+0000", 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5698999999999995e-05 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885805+0000", 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0442000000000001e-05 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34114 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 851968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.545088163000003, 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0734813590000001, 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34114 2024-05-06T00:28:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-07 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812540+0000", 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4509999999999999e-06 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5069999999999997e-06 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812552+0000", 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817936+0000", 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053840650000000004 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817944+0000", 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5890000000000003e-06 2024-05-06T00:28:42.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885879+0000", 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0679342119999999 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885952+0000", 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2924999999999997e-05 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886019+0000", 2024-05-06T00:28:42.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7358000000000001e-05 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34115 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 858112~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.545053054, 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0736428419999999, 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34115 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9700000000000002e-07 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812575+0000", 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.402e-06 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:42.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812587+0000", 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7979999999999996e-06 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817971+0000", 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053842719999999998 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817979+0000", 2024-05-06T00:28:42.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4250000000000001e-06 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886063+0000", 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068084475 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886129+0000", 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6109e-05 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886216+0000", 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6312999999999998e-05 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.387 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.545020063999999, 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:28:42.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:28:42.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544988693999997, 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:28:42.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:28:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544954904999997, 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:28:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:28:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544774449000002, 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:28:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:28:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544740806, 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:28:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544717024000001, 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:28:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:28:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544690746999997, 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:28:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:28:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544647505999997, 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:28:42.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:28:42.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:28:42.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544617877, 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:28:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544585026999997, 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:42.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:28:42.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:28:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544553389000001, 2024-05-06T00:28:42.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:28:42.444 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.444 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:28:42.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.445 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:42.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:42.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:28:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:28:42.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:28:42.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:28:42.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:28:42.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.451 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.451 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.451 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:42.451 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544520587000001, 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.452 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:28:42.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:28:42.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:28:42.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:28:42.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:28:42.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:28:42.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:28:42.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:28:42.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:28:42.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:28:42.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.459 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.460 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:42.460 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544488553999997, 2024-05-06T00:28:42.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:28:42.460 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.460 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.460 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:28:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:28:42.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:28:42.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:28:42.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:28:42.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:28:42.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:28:42.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:28:42.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:28:42.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:28:42.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:28:42.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:28:42.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:28:42.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:28:42.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:28:42.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.468 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:42.469 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544456474999997, 2024-05-06T00:28:42.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:28:42.469 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:28:42.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.470 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:42.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:28:42.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:28:42.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:42.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:28:42.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:28:42.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:28:42.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:28:42.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:42.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:28:42.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:28:42.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:28:42.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:28:42.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:28:42.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:28:42.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:28:42.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:28:42.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.477 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.544425167, 2024-05-06T00:28:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:28:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:28:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:42.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:42.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:42.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:42.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:42.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:42.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:28:42.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:28:42.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:42.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:28:42.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:28:42.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:42.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:28:42.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:28:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:28:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:28:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:42.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:28:42.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:28:42.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:42.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:28:42.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:28:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:28:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:28:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.486 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:42.486 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:42.486 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:42.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:42.489 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:42.485+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:42.500 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 0, 5, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:42.501 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:42.501 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-05-06T00:28:42.501 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-06T00:28:42.501 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-06T00:28:42.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-05-06T00:28:42.596 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:42.593+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:42.673 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:42.673 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "" 2024-05-06T00:28:42.674 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:42.675 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:42.675 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:42.676 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:42.676 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:42.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:28:42.690 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:42.686+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:42.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:42.791 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:42.789+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:42.804 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone28 from 0x6399000 (overlap 0x1d4000) to 0xea000 2024-05-06T00:28:42.865 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:42.865 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:42.865 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:42.865 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:42.865 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:42.879 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:42.892 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:42.886+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:42.992 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:42.986+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21566 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 417792~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017916006999997, 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090303018999999998, 2024-05-06T00:28:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21566 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5400000000000002e-07 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026960+0000", 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4449999999999999e-06 2024-05-06T00:28:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026966+0000", 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6200000000000001e-06 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026978+0000", 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1248e-05 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116885+0000", 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.089907163999999998 2024-05-06T00:28:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116986+0000", 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101363 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117255+0000", 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026882499999999999 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21568 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 423936~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017808522000003, 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090403081999999996, 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21568 2024-05-06T00:28:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027060+0000", 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027063+0000", 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3249999999999999e-06 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027072+0000", 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2859999999999999e-06 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027078+0000", 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117286+0000", 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090207976999999995 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117372+0000", 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5865000000000005e-05 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117463+0000", 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0662999999999996e-05 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21570 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 430080~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017781913, 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090545366000000002, 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21570 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027087+0000", 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3799999999999998e-07 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027089+0000", 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.12e-06 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027095+0000", 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4590000000000003e-06 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027100+0000", 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117492+0000", 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090391632999999999 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117546+0000", 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4177e-05 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117631+0000", 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5173999999999995e-05 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21572 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 436224~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017759552000001, 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090693918999999998, 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21572 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027109+0000", 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7100000000000002e-07 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027112+0000", 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5490000000000001e-06 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027119+0000", 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7829999999999999e-06 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027124+0000", 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.04e-06 2024-05-06T00:28:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117662+0000", 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090537113000000002 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117717+0000", 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5798999999999999e-05 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117802+0000", 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5063999999999997e-05 2024-05-06T00:28:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017735485999999, 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017714697000002, 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017694634999998, 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:28:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:28:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017675359000002, 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017655241, 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017623831999998, 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:28:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017590920000004, 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:28:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017570333000002, 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:28:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:28:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:28:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017550952999997, 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017506271999999, 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:28:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:28:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017441181000002, 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:28:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017409606999998, 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:28:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:28:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:28:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.017387346, 2024-05-06T00:28:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:28:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.01736623, 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 38.359649603999998, 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:28:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:28:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 38.359239361999997, 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:43.134 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:43.135 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:43.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:43.137 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:43.130+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:43.236 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:43.237+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:43.278 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-05-06T00:28:43.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:43.353 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:43.369 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:43.369+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:43.439 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:43.437+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:43.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:43.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:43.517 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:43.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:43.540 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:43.534+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:43.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:43.553 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:43.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:43.576 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:43.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:43.598 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:43.618 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:43.638 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:43.639 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:43.641 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:43.641+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:43.653 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:43.669 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:28:43.742 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:43.738+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:43.795 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:43.795 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:43.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:43.853 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:43.850+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:43.944 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:43.938+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:43.956 INFO:teuthology.orchestra.run.smithi186.stdout:4500 read 0x2ff9e00 thru 0x300336e (0x956f bytes) 2024-05-06T00:28:43.957 INFO:teuthology.orchestra.run.smithi186.stdout:4502 read 0x3615a00 thru 0x3621a68 (0xc069 bytes) 2024-05-06T00:28:43.959 INFO:teuthology.orchestra.run.smithi186.stdout:4504 punch from 0x49c7800 to 0x49c8a00, (0x1200 bytes) 2024-05-06T00:28:43.974 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:43.975 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:43.976 INFO:teuthology.orchestra.run.smithi186.stdout:4507 read 0x35f0600 thru 0x35f7335 (0x6d36 bytes) 2024-05-06T00:28:43.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:44.025 INFO:teuthology.orchestra.run.smithi186.stdout:4509 write 0xad61000 thru 0xad6eec5 (0xdec6 bytes) 2024-05-06T00:28:44.026 INFO:teuthology.orchestra.run.smithi186.stdout:4515 read 0x2ce5600 thru 0x2cf1abc (0xc4bd bytes) 2024-05-06T00:28:44.026 INFO:teuthology.orchestra.run.smithi186.stdout:4516 read 0x1bbbc00 thru 0x1bc8807 (0xcc08 bytes) 2024-05-06T00:28:44.027 INFO:teuthology.orchestra.run.smithi186.stdout:4518 write 0x8260800 thru 0x8263af4 (0x32f5 bytes) 2024-05-06T00:28:44.034 INFO:teuthology.orchestra.run.smithi186.stdout:4520 trunc from 0xad6f000 to 0xbe6b600 2024-05-06T00:28:44.045 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:44.045+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:44.080 INFO:teuthology.orchestra.run.smithi186.stdout:4524 trunc from 0xbe6b600 to 0xdf64a00 2024-05-06T00:28:44.104 INFO:teuthology.orchestra.run.smithi186.stdout:4525 read 0xb956c00 thru 0xb95e096 (0x7497 bytes) 2024-05-06T00:28:44.107 INFO:teuthology.orchestra.run.smithi186.stdout:4527 write 0x2e2c200 thru 0x2e32e27 (0x6c28 bytes) 2024-05-06T00:28:44.123 INFO:teuthology.orchestra.run.smithi186.stdout:4529 trunc from 0xdf64a00 to 0x66ad000 2024-05-06T00:28:44.146 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:44.145+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:44.158 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:44.158 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:44.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:44.204 INFO:teuthology.orchestra.run.smithi186.stdout:4530 write 0x3e56200 thru 0x3e64817 (0xe618 bytes) 2024-05-06T00:28:44.208 INFO:teuthology.orchestra.run.smithi186.stdout:4531 read 0x2bcc400 thru 0x2bd8d52 (0xc953 bytes) 2024-05-06T00:28:44.209 INFO:teuthology.orchestra.run.smithi186.stdout:4532 read 0x6515000 thru 0x6520c42 (0xbc43 bytes) 2024-05-06T00:28:44.210 INFO:teuthology.orchestra.run.smithi186.stdout:4533 punch from 0x611b800 to 0x6127a00, (0xc200 bytes) 2024-05-06T00:28:44.211 INFO:teuthology.orchestra.run.smithi186.stdout:4535 clone 30 order 21 su 65536 sc 6 2024-05-06T00:28:44.247 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:44.245+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:44.300 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:44.300 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:44.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:44.349 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:44.349+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:44.437 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:44.437 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:44.450 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:44.449+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:44.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:44.550 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:44.546+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:44.573 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:44.574 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:44.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:44.677 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:44.674+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:44.752 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:44.746+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:44.757 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:44.758 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:44.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:44.861 INFO:teuthology.orchestra.run.smithi186.stdout:leaving image image_client.0-clone29 intact 2024-05-06T00:28:44.877 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:44.874+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:44.952 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:44.952 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:44.954 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:44.950+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:44.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:45.055 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:45.050+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:45.129 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:45.129 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:45.143 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:45.185 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:45.182+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:45.257 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:45.254+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:45.279 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:45.279 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:45.293 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:45.358 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:45.357+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:45.419 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:45.419 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:45.434 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:45.451 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-05-06T00:28:45.476 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:45.477+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:45.527 INFO:teuthology.orchestra.run.smithi186.stdout:4537 read 0x4c28a00 thru 0x4c36d50 (0xe351 bytes) 2024-05-06T00:28:45.529 INFO:teuthology.orchestra.run.smithi186.stdout:4538 punch from 0x5dbf200 to 0x5dc2200, (0x3000 bytes) 2024-05-06T00:28:45.549 INFO:teuthology.orchestra.run.smithi186.stdout:4539 read 0x45c9800 thru 0x45d1f54 (0x8755 bytes) 2024-05-06T00:28:45.550 INFO:teuthology.orchestra.run.smithi186.stdout:4540 write 0x2953e00 thru 0x29632f8 (0xf4f9 bytes) 2024-05-06T00:28:45.551 INFO:teuthology.orchestra.run.smithi186.stdout:4542 write 0x508ae00 thru 0x508eb7f (0x3d80 bytes) 2024-05-06T00:28:45.560 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:45.557+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:45.606 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:45.606 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:45.621 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:45.621 INFO:teuthology.orchestra.run.smithi186.stdout:4543 write 0x9b41c00 thru 0x9b457aa (0x3bab bytes) 2024-05-06T00:28:45.622 INFO:teuthology.orchestra.run.smithi186.stdout:4544 read 0x263cc00 thru 0x26451c4 (0x85c5 bytes) 2024-05-06T00:28:45.622 INFO:teuthology.orchestra.run.smithi186.stdout:4547 read 0x162c400 thru 0x1638431 (0xc032 bytes) 2024-05-06T00:28:45.623 INFO:teuthology.orchestra.run.smithi186.stdout:4551 read 0x7164200 thru 0x7165df7 (0x1bf8 bytes) 2024-05-06T00:28:45.623 INFO:teuthology.orchestra.run.smithi186.stdout:4552 write 0x821c400 thru 0x8225713 (0x9314 bytes) 2024-05-06T00:28:45.624 INFO:teuthology.orchestra.run.smithi186.stdout:4555 punch from 0x95f7a00 to 0x95fe000, (0x6600 bytes) 2024-05-06T00:28:45.624 INFO:teuthology.orchestra.run.smithi186.stdout:4556 write 0x1015a00 thru 0x101f85d (0x9e5e bytes) 2024-05-06T00:28:45.626 INFO:teuthology.orchestra.run.smithi186.stdout:4557 write 0x6171600 thru 0x617f619 (0xe01a bytes) 2024-05-06T00:28:45.627 INFO:teuthology.orchestra.run.smithi186.stdout:4558 write 0x3ce6800 thru 0x3cebd5f (0x5560 bytes) 2024-05-06T00:28:45.628 INFO:teuthology.orchestra.run.smithi186.stdout:4559 read 0x2477600 thru 0x24862b3 (0xecb4 bytes) 2024-05-06T00:28:45.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:45.637 INFO:teuthology.orchestra.run.smithi186.stdout:4561 punch from 0x944de00 to 0x9450400, (0x2600 bytes) 2024-05-06T00:28:45.639 INFO:teuthology.orchestra.run.smithi186.stdout:4565 read 0x8561400 thru 0x85670ef (0x5cf0 bytes) 2024-05-06T00:28:45.639 INFO:teuthology.orchestra.run.smithi186.stdout:4566 write 0x4237e00 thru 0x423fa93 (0x7c94 bytes) 2024-05-06T00:28:45.665 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:45.665+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:45.669 INFO:teuthology.orchestra.run.smithi186.stdout:4567 trunc from 0x9b45800 to 0x47c4e00 2024-05-06T00:28:45.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:28:45.723 INFO:teuthology.orchestra.run.smithi186.stdout:4569 punch from 0x17d8800 to 0x17e3400, (0xac00 bytes) 2024-05-06T00:28:45.759 INFO:teuthology.orchestra.run.smithi186.stdout:4571 write 0xa1ca800 thru 0xa1cb6ea (0xeeb bytes) 2024-05-06T00:28:45.761 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:45.758+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:45.799 INFO:teuthology.orchestra.run.smithi186.stdout:4573 trunc from 0xa1cb800 to 0xcba9a00 2024-05-06T00:28:45.799 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:45.799 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:45.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:45.816 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:45.816 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:45.822 INFO:teuthology.orchestra.run.smithi186.stdout:4576 write 0x793bc00 thru 0x7947bab (0xbfac bytes) 2024-05-06T00:28:45.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:45.835 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:45.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:45.855 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:45.862 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:45.858+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:45.867 INFO:teuthology.orchestra.run.smithi186.stdout:4577 trunc from 0xcba9a00 to 0x2c5ee00 2024-05-06T00:28:45.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:45.892 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:45.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:45.932 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:45.932 INFO:teuthology.orchestra.run.smithi186.stdout:4578 read 0x592200 thru 0x5a1a12 (0xf813 bytes) 2024-05-06T00:28:45.934 INFO:teuthology.orchestra.run.smithi186.stdout:4582 read 0x1a38e00 thru 0x1a3af05 (0x2106 bytes) 2024-05-06T00:28:45.935 INFO:teuthology.orchestra.run.smithi186.stdout:4584 read 0x244d800 thru 0x245c2b2 (0xeab3 bytes) 2024-05-06T00:28:45.937 INFO:teuthology.orchestra.run.smithi186.stdout:4585 punch from 0x1746c00 to 0x1756800, (0xfc00 bytes) 2024-05-06T00:28:45.963 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:45.961+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:45.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:45.966 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:45.968 INFO:teuthology.orchestra.run.smithi186.stdout:4587 write 0x7712000 thru 0x771f6a2 (0xd6a3 bytes) 2024-05-06T00:28:45.969 INFO:teuthology.orchestra.run.smithi186.stdout:4588 read 0x29d8800 thru 0x29e0df3 (0x85f4 bytes) 2024-05-06T00:28:45.971 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:45.972 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:45.983 INFO:teuthology.orchestra.run.smithi186.stdout:4590 trunc from 0x771f800 to 0xbe93800 2024-05-06T00:28:45.988 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:46.039 INFO:teuthology.orchestra.run.smithi186.stdout:4591 write 0xdceb600 thru 0xdcf64d9 (0xaeda bytes) 2024-05-06T00:28:46.042 INFO:teuthology.orchestra.run.smithi186.stdout:4592 read 0x6953400 thru 0x6959220 (0x5e21 bytes) 2024-05-06T00:28:46.057 INFO:teuthology.orchestra.run.smithi186.stdout:4594 trunc from 0xdcf6600 to 0x52e2a00 2024-05-06T00:28:46.065 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:46.058+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:46.090 INFO:teuthology.orchestra.run.smithi186.stdout:4596 read 0x1bf6600 thru 0x1bfaf19 (0x491a bytes) 2024-05-06T00:28:46.109 INFO:teuthology.orchestra.run.smithi186.stdout:4603 write 0x6b55200 thru 0x6b57376 (0x2177 bytes) 2024-05-06T00:28:46.117 INFO:teuthology.orchestra.run.smithi186.stdout:4604 trunc from 0x6b57400 to 0x30d8c00 2024-05-06T00:28:46.150 INFO:teuthology.orchestra.run.smithi186.stdout:4605 read 0x1fb6800 thru 0x1fba984 (0x4185 bytes) 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21566 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 417792~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125268564999999, 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090303018999999998, 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21566 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026952+0000", 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5400000000000002e-07 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026960+0000", 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4449999999999999e-06 2024-05-06T00:28:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026966+0000", 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6200000000000001e-06 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.026978+0000", 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1248e-05 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116885+0000", 2024-05-06T00:28:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.089907163999999998 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.116986+0000", 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101363 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117255+0000", 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026882499999999999 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21568 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 423936~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125161079999998, 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090403081999999996, 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21568 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027059+0000", 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027060+0000", 2024-05-06T00:28:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027063+0000", 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3249999999999999e-06 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027072+0000", 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2859999999999999e-06 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027078+0000", 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117286+0000", 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090207976999999995 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117372+0000", 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5865000000000005e-05 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117463+0000", 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0662999999999996e-05 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21570 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 430080~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125134471000003, 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090545366000000002, 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21570 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027086+0000", 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027087+0000", 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3799999999999998e-07 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027089+0000", 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.12e-06 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027095+0000", 2024-05-06T00:28:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4590000000000003e-06 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027100+0000", 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117492+0000", 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090391632999999999 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117546+0000", 2024-05-06T00:28:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4177e-05 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117631+0000", 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5173999999999995e-05 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21572 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 436224~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125112110000003, 2024-05-06T00:28:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090693918999999998, 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21572 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027108+0000", 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027109+0000", 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7100000000000002e-07 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027112+0000", 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5490000000000001e-06 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027119+0000", 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7829999999999999e-06 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027124+0000", 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.04e-06 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117662+0000", 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090537113000000002 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117717+0000", 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5798999999999999e-05 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117802+0000", 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5063999999999997e-05 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125088044000002, 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:28:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125067254999998, 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:28:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125047193, 2024-05-06T00:28:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:28:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125027916999997, 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.125007799000002, 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:28:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:28:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.12497639, 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:28:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.124943477999999, 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:28:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.124922890999997, 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.124903510999999, 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:28:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:28:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.124858830000001, 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:28:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.124793738999998, 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:28:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:28:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.124762165, 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.124739904000002, 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:28:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:28:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.124718788000003, 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:28:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.467002162, 2024-05-06T00:28:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:28:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.466591919999999, 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.238 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:46.240 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:46.240 INFO:teuthology.orchestra.run.smithi186.stdout:4606 trunc from 0x30d8c00 to 0x641c800 2024-05-06T00:28:46.240 INFO:teuthology.orchestra.run.smithi186.stdout:4607 punch from 0x299d400 to 0x29a3c00, (0x6800 bytes) 2024-05-06T00:28:46.241 INFO:teuthology.orchestra.run.smithi186.stdout:4609 read 0x24de800 thru 0x24e201e (0x381f bytes) 2024-05-06T00:28:46.241 INFO:teuthology.orchestra.run.smithi186.stdout:4612 write 0x5f22c00 thru 0x5f2cd06 (0xa107 bytes) 2024-05-06T00:28:46.241 INFO:teuthology.orchestra.run.smithi186.stdout:4613 write 0xe046400 thru 0xe04a8f8 (0x44f9 bytes) 2024-05-06T00:28:46.241 INFO:teuthology.orchestra.run.smithi186.stdout:4618 punch from 0x3957000 to 0x3964e00, (0xde00 bytes) 2024-05-06T00:28:46.241 INFO:teuthology.orchestra.run.smithi186.stdout:4619 write 0x958c000 thru 0x959170e (0x570f bytes) 2024-05-06T00:28:46.241 INFO:teuthology.orchestra.run.smithi186.stdout:4621 read 0x3aec600 thru 0x3af267c (0x607d bytes) 2024-05-06T00:28:46.241 INFO:teuthology.orchestra.run.smithi186.stdout:4622 read 0x7e51200 thru 0x7e5ce0a (0xbc0b bytes) 2024-05-06T00:28:46.241 INFO:teuthology.orchestra.run.smithi186.stdout:4624 trunc from 0xe04aa00 to 0xdfc3800 2024-05-06T00:28:46.242 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:46.234+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:46.295 INFO:teuthology.orchestra.run.smithi186.stdout:4625 trunc from 0xdfc3800 to 0x1587600 2024-05-06T00:28:46.339 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:46.337+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:46.363 INFO:teuthology.orchestra.run.smithi186.stdout:4626 write 0x1fe1200 thru 0x1fe684e (0x564f bytes) 2024-05-06T00:28:46.364 INFO:teuthology.orchestra.run.smithi186.stdout:4627 punch from 0x115d400 to 0x1161600, (0x4200 bytes) 2024-05-06T00:28:46.375 INFO:teuthology.orchestra.run.smithi186.stdout:4630 trunc from 0x1fe6a00 to 0x7b40400 2024-05-06T00:28:46.382 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:46.383 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:46.393 INFO:teuthology.orchestra.run.smithi186.stdout:4631 punch from 0x183d200 to 0x1841400, (0x4200 bytes) 2024-05-06T00:28:46.396 INFO:teuthology.orchestra.run.smithi186.stdout:4636 write 0x4aeb800 thru 0x4af32b6 (0x7ab7 bytes) 2024-05-06T00:28:46.396 INFO:teuthology.orchestra.run.smithi186.stdout:4637 read 0x110c800 thru 0x1115241 (0x8a42 bytes) 2024-05-06T00:28:46.397 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:46.398 INFO:teuthology.orchestra.run.smithi186.stdout:truncating to largest ever: 0xee12800 2024-05-06T00:28:46.412 INFO:teuthology.orchestra.run.smithi186.stdout:4638 trunc from 0x7b40400 to 0xee12800 2024-05-06T00:28:46.440 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:46.438+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:46.441 INFO:teuthology.orchestra.run.smithi186.stdout:4642 write 0x2dad600 thru 0x2dbad08 (0xd709 bytes) 2024-05-06T00:28:46.442 INFO:teuthology.orchestra.run.smithi186.stdout:4643 punch from 0xbde9a00 to 0xbdf4e00, (0xb400 bytes) 2024-05-06T00:28:46.445 INFO:teuthology.orchestra.run.smithi186.stdout:4644 trunc from 0xee12800 to 0xdae3c00 2024-05-06T00:28:46.473 INFO:teuthology.orchestra.run.smithi186.stdout:4647 punch from 0x647bc00 to 0x647ca00, (0xe00 bytes) 2024-05-06T00:28:46.489 INFO:teuthology.orchestra.run.smithi186.stdout:4648 trunc from 0xdae3c00 to 0x5af9400 2024-05-06T00:28:46.514 INFO:teuthology.orchestra.run.smithi186.stdout:4649 read 0x2a0c200 thru 0x2a0cb0a (0x90b bytes) 2024-05-06T00:28:46.516 INFO:teuthology.orchestra.run.smithi186.stdout:4650 read 0x1bd3200 thru 0x1bd4bd7 (0x19d8 bytes) 2024-05-06T00:28:46.516 INFO:teuthology.orchestra.run.smithi186.stdout:4651 read 0x1207400 thru 0x120a7bd (0x33be bytes) 2024-05-06T00:28:46.517 INFO:teuthology.orchestra.run.smithi186.stdout:4652 read 0x1a8e00 thru 0x1b3bdf (0xade0 bytes) 2024-05-06T00:28:46.521 INFO:teuthology.orchestra.run.smithi186.stdout:4655 trunc from 0x5af9400 to 0x72dee00 2024-05-06T00:28:46.538 INFO:teuthology.orchestra.run.smithi186.stdout:4656 trunc from 0x72dee00 to 0x72d0800 2024-05-06T00:28:46.541 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:46.541+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:46.561 INFO:teuthology.orchestra.run.smithi186.stdout:4658 punch from 0x5d83c00 to 0x5d86c00, (0x3000 bytes) 2024-05-06T00:28:46.572 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:46.572 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:46.587 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:46.590 INFO:teuthology.orchestra.run.smithi186.stdout:4665 write 0xc384000 thru 0xc3921f0 (0xe1f1 bytes) 2024-05-06T00:28:46.593 INFO:teuthology.orchestra.run.smithi186.stdout:4666 read 0x44a4c00 thru 0x44b1ef7 (0xd2f8 bytes) 2024-05-06T00:28:46.593 INFO:teuthology.orchestra.run.smithi186.stdout:4667 write 0x2d9e000 thru 0x2dad41d (0xf41e bytes) 2024-05-06T00:28:46.594 INFO:teuthology.orchestra.run.smithi186.stdout:4672 write 0x23fa00 thru 0x247c95 (0x8296 bytes) 2024-05-06T00:28:46.624 INFO:teuthology.orchestra.run.smithi186.stdout:4673 trunc from 0xc392200 to 0x3cf800 2024-05-06T00:28:46.672 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:46.673+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:46.691 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:28:46.742 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:46.738+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:46.762 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:46.762 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:46.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:46.762 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.756631561000006, 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:28:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:28:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.756402988999994, 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:28:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:28:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.756325704999995, 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:28:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:28:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.756129724000004, 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:28:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.849020348000003, 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.848959469, 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.731403018999998, 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:28:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.731358290999999, 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.731319890000002, 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:28:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.731282628000002, 2024-05-06T00:28:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:28:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.731245719, 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:28:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:28:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.731207800999996, 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:28:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:28:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.731170685999999, 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.725390236999999, 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.725352153000003, 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.725314478999998, 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:28:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.715878969000002, 2024-05-06T00:28:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:28:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.715846542000001, 2024-05-06T00:28:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:28:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:28:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.715822660000001, 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:28:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.71580273, 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:28:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:28:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:46.850 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:46.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:46.852 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:46.846+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:46.953 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:46.953+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:46.982 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:46.982 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:46.982 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:46.982 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:46.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:47.007 INFO:teuthology.orchestra.run.smithi186.stdout:4675 write 0xec30600 thru 0xec3124b (0xc4c bytes) 2024-05-06T00:28:47.009 INFO:teuthology.orchestra.run.smithi186.stdout:4677 punch from 0x753e800 to 0x7542e00, (0x4600 bytes) 2024-05-06T00:28:47.009 INFO:teuthology.orchestra.run.smithi186.stdout:4678 read 0x65fe200 thru 0x660211d (0x3f1e bytes) 2024-05-06T00:28:47.010 INFO:teuthology.orchestra.run.smithi186.stdout:4679 write 0x6a56400 thru 0x6a60343 (0x9f44 bytes) 2024-05-06T00:28:47.010 INFO:teuthology.orchestra.run.smithi186.stdout:4682 read 0x78c1c00 thru 0x78c3c80 (0x2081 bytes) 2024-05-06T00:28:47.010 INFO:teuthology.orchestra.run.smithi186.stdout:4685 write 0x8ea2600 thru 0x8eaa2a0 (0x7ca1 bytes) 2024-05-06T00:28:47.011 INFO:teuthology.orchestra.run.smithi186.stdout:4686 read 0xa78a000 thru 0xa78c2c3 (0x22c4 bytes) 2024-05-06T00:28:47.011 INFO:teuthology.orchestra.run.smithi186.stdout:4687 read 0x3531200 thru 0x3534baa (0x39ab bytes) 2024-05-06T00:28:47.011 INFO:teuthology.orchestra.run.smithi186.stdout:4689 punch from 0x22fcc00 to 0x2309200, (0xc600 bytes) 2024-05-06T00:28:47.011 INFO:teuthology.orchestra.run.smithi186.stdout:4690 write 0xe1ea800 thru 0xe1f9319 (0xeb1a bytes) 2024-05-06T00:28:47.021 INFO:teuthology.orchestra.run.smithi186.stdout:4692 trunc from 0xec31400 to 0xa16fc00 2024-05-06T00:28:47.054 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:47.050+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:47.058 INFO:teuthology.orchestra.run.smithi186.stdout:4694 write 0x7460600 thru 0x7466b9d (0x659e bytes) 2024-05-06T00:28:47.060 INFO:teuthology.orchestra.run.smithi186.stdout:4695 read 0x484c00 thru 0x489d04 (0x5105 bytes) 2024-05-06T00:28:47.061 INFO:teuthology.orchestra.run.smithi186.stdout:4699 write 0x69bba00 thru 0x69ca198 (0xe799 bytes) 2024-05-06T00:28:47.061 INFO:teuthology.orchestra.run.smithi186.stdout:4700 read 0x67d0000 thru 0x67dee7a (0xee7b bytes) 2024-05-06T00:28:47.061 INFO:teuthology.orchestra.run.smithi186.stdout:4702 punch from 0x9ecd200 to 0x9ed3200, (0x6000 bytes) 2024-05-06T00:28:47.062 INFO:teuthology.orchestra.run.smithi186.stdout:4705 write 0x22f6600 thru 0x2306116 (0xfb17 bytes) 2024-05-06T00:28:47.095 INFO:teuthology.orchestra.run.smithi186.stdout:4707 write 0xc188800 thru 0xc18fc4e (0x744f bytes) 2024-05-06T00:28:47.097 INFO:teuthology.orchestra.run.smithi186.stdout:4708 read 0x2dbc00 thru 0x2df28f (0x3690 bytes) 2024-05-06T00:28:47.098 INFO:teuthology.orchestra.run.smithi186.stdout:4711 read 0x5f92e00 thru 0x5f9c060 (0x9261 bytes) 2024-05-06T00:28:47.111 INFO:teuthology.orchestra.run.smithi186.stdout:4713 trunc from 0xc18fe00 to 0x73d8e00 2024-05-06T00:28:47.125 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:47.125 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:47.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:47.155 INFO:teuthology.orchestra.run.smithi186.stdout:4715 punch from 0x1722e00 to 0x1728000, (0x5200 bytes) 2024-05-06T00:28:47.159 INFO:teuthology.orchestra.run.smithi186.stdout:4717 trunc from 0x73d8e00 to 0x7ba2200 2024-05-06T00:28:47.189 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:47.186+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:47.209 INFO:teuthology.orchestra.run.smithi186.stdout:4718 write 0x7ccd000 thru 0x7cd6383 (0x9384 bytes) 2024-05-06T00:28:47.210 INFO:teuthology.orchestra.run.smithi186.stdout:4720 punch from 0x3982600 to 0x398f000, (0xca00 bytes) 2024-05-06T00:28:47.211 INFO:teuthology.orchestra.run.smithi186.stdout:4722 punch from 0x71dd600 to 0x71e9200, (0xbc00 bytes) 2024-05-06T00:28:47.211 INFO:teuthology.orchestra.run.smithi186.stdout:4723 punch from 0x4e1c000 to 0x4e21a00, (0x5a00 bytes) 2024-05-06T00:28:47.211 INFO:teuthology.orchestra.run.smithi186.stdout:4727 read 0x4955e00 thru 0x495f989 (0x9b8a bytes) 2024-05-06T00:28:47.211 INFO:teuthology.orchestra.run.smithi186.stdout:4728 read 0x1d01200 thru 0x1d06a90 (0x5891 bytes) 2024-05-06T00:28:47.212 INFO:teuthology.orchestra.run.smithi186.stdout:4729 write 0x1f8f800 thru 0x1f9703f (0x7840 bytes) 2024-05-06T00:28:47.213 INFO:teuthology.orchestra.run.smithi186.stdout:4732 punch from 0x395000 to 0x39e200, (0x9200 bytes) 2024-05-06T00:28:47.225 INFO:teuthology.orchestra.run.smithi186.stdout:4733 trunc from 0x7cd6400 to 0xd6d0c00 2024-05-06T00:28:47.248 INFO:teuthology.orchestra.run.smithi186.stdout:4734 read 0xaa9000 thru 0xaa9044 (0x45 bytes) 2024-05-06T00:28:47.250 INFO:teuthology.orchestra.run.smithi186.stdout:4736 write 0xc69200 thru 0xc75a74 (0xc875 bytes) 2024-05-06T00:28:47.250 INFO:teuthology.orchestra.run.smithi186.stdout:4738 write 0x1ba1600 thru 0x1baa4ed (0x8eee bytes) 2024-05-06T00:28:47.252 INFO:teuthology.orchestra.run.smithi186.stdout:4741 trunc from 0xd6d0c00 to 0xc69aa00 2024-05-06T00:28:47.256 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:47.250+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:47.286 INFO:teuthology.orchestra.run.smithi186.stdout:4742 write 0x5af4200 thru 0x5afe224 (0xa025 bytes) 2024-05-06T00:28:47.290 INFO:teuthology.orchestra.run.smithi186.stdout:4744 trunc from 0xc69aa00 to 0xcb5a400 2024-05-06T00:28:47.323 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:47.323 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 137.71949168899999, 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:28:47.324 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:28:47.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:28:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.330 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446842989999993, 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:28:47.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:28:47.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:28:47.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446807813999996, 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.336 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:28:47.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.341 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446719134999995, 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.342 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:28:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446617192000005, 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:28:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446508089999995, 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:28:47.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:28:47.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446410231000002, 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:28:47.359 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:28:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:28:47.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:28:47.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446345180999998, 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:28:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:28:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:28:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446252534999999, 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:28:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.446180530999996, 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:28:47.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.385 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686719045000004, 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.386 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:28:47.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686699797000003, 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:28:47.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:28:47.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686649677000005, 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:28:47.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:28:47.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.403 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686628626000001, 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686613307000002, 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.409 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:28:47.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:28:47.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:28:47.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686596425000005, 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.415 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:28:47.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686578088999994, 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.421 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:47.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686562086999999, 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:28:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:28:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.686475724000005, 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:28:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.676513248000006, 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:28:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:28:47.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi186.stdout:4745 write 0x8a16800 thru 0x8a249ba (0xe1bb bytes) 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi186.stdout:4746 read 0x289d400 thru 0x28a30cc (0x5ccd bytes) 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi186.stdout:4747 read 0x849c800 thru 0x84a7db6 (0xb5b7 bytes) 2024-05-06T00:28:47.444 INFO:teuthology.orchestra.run.smithi186.stdout:4748 punch from 0x9361400 to 0x936d400, (0xc000 bytes) 2024-05-06T00:28:47.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:4749 trunc from 0xcb5a400 to 0x2ee7200 2024-05-06T00:28:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:4751 punch from 0x209400 to 0x217400, (0xe000 bytes) 2024-05-06T00:28:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:4755 read 0x17c5800 thru 0x17c9613 (0x3e14 bytes) 2024-05-06T00:28:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:4756 read 0xc0ac00 thru 0xc0dfb5 (0x33b6 bytes) 2024-05-06T00:28:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:4758 write 0x9b48800 thru 0x9b4c32f (0x3b30 bytes) 2024-05-06T00:28:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:4759 read 0x1d7e00 thru 0x1e65c2 (0xe7c3 bytes) 2024-05-06T00:28:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:4760 read 0x6c5e00 thru 0x6cf629 (0x982a bytes) 2024-05-06T00:28:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:4761 punch from 0x51b5000 to 0x51c3200, (0xe200 bytes) 2024-05-06T00:28:47.448 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:47.442+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:47.472 INFO:teuthology.orchestra.run.smithi186.stdout:4762 write 0xb174800 thru 0xb177690 (0x2e91 bytes) 2024-05-06T00:28:47.474 INFO:teuthology.orchestra.run.smithi186.stdout:4763 punch from 0x1678200 to 0x167cc00, (0x4a00 bytes) 2024-05-06T00:28:47.474 INFO:teuthology.orchestra.run.smithi186.stdout:4767 write 0x2767000 thru 0x2774c5f (0xdc60 bytes) 2024-05-06T00:28:47.475 INFO:teuthology.orchestra.run.smithi186.stdout:4768 punch from 0x2c76a00 to 0x2c78800, (0x1e00 bytes) 2024-05-06T00:28:47.475 INFO:teuthology.orchestra.run.smithi186.stdout:4770 read 0xa066400 thru 0xa06bd7d (0x597e bytes) 2024-05-06T00:28:47.475 INFO:teuthology.orchestra.run.smithi186.stdout:4771 read 0x2847400 thru 0x284c7ce (0x53cf bytes) 2024-05-06T00:28:47.476 INFO:teuthology.orchestra.run.smithi186.stdout:4772 clone 31 order 22 su 131072 sc 15 2024-05-06T00:28:47.546 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:47.545+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:47.618 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:47.618 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:47.618 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:47.618 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:47.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:47.681 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:47.678+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:47.690 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 0, 5, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:47.690 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:47.690 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 4, 0, 5, 3] 2024-05-06T00:28:47.691 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-05-06T00:28:47.748 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:47.742+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:47.756 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:47.756 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:47.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:47.848 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:47.849+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:47.880 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone30 from 0xb177800 (overlap 0x3cf800) to 0x249400 2024-05-06T00:28:47.949 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:47.949+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:47.955 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:47.955 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:47.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:47.955 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:47.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.295473451000007, 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:28:47.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:28:47.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.316231946000002, 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.961 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:28:47.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.316157726999997, 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:28:47.967 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:28:47.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.316095828000002, 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:28:47.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:28:47.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:47.976 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.316036228999998, 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:28:47.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.981 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315952093999996, 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.982 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:28:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315886581000001, 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.987 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:47.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.991 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315828719000002, 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:47.992 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:28:47.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:47.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315732124, 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:47.997 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:47.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315654399000003, 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:28:48.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315581612999999, 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.006 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:48.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.010 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315504122, 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315409993000003, 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:28:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:28:48.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.020 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315327031000002, 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.021 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:28:48.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:28:48.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.026 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.026 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.027 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:48.027 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315252925999999, 2024-05-06T00:28:48.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:28:48.027 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.027 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.027 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.028 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:28:48.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.029 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:48.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:28:48.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315153737000003, 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.034 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:28:48.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:28:48.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.038 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.315090585, 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:28:48.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:28:48.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.31499032, 2024-05-06T00:28:48.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:28:48.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.314900948000002, 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.048 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:28:48.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:28:48.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.069042265, 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:48.053 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:48.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:28:48.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:48.058 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:48.059 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:48.060 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-06T00:28:48.029+0000 7fdfb36a9640 -1 mon.a@0(leader).osd e116 definitely_dead 0 2024-05-06T00:28:48.061 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:48.061+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:48.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:48.161 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:48.158+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:48.193 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:48.193 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:48.193 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:48.193 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:48.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:48.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:48.208 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:48.225 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:48.225 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:48.242 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:48.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:48.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:48.280 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:48.293 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:48.290+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:48.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:48.363 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:48.358+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:48.368 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:48.368 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:48.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:48.462 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-05-06T00:28:48.473 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:48.470+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:48.558 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:48.558 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:48.564 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:48.561+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:48.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:48.665 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:48.665+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.2. 2024-05-06T00:28:48.714 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:28:48.724 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:48.724 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:48.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:48.788 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:48.789+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:48.867 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:48.862+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:48.904 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:48.904 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:48.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:48.968 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:48.962+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:49.035 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:49.035 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:49.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:49.096 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:49.097+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:49.169 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:49.169+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:49.173 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:49.173 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:49.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:49.270 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:49.270+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:49.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:49.371 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:49.366+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:49.472 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:49.466+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:49.484 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:49.484 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:49.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:49.573 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:49.570+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:49.621 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:49.622 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:49.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:49.674 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:49.670+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:49.704 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:28:49.761 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:49.763 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:49.774 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:49.773+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:49.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:49.875 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:49.873+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:49.939 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:49.939 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:49.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:50.001 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:49.998+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:50.077 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:50.077+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:50.085 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:50.086 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:50.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:50.178 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:50.177+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:50.261 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:50.261 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:50.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:50.279 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:50.274+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:50.320 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:50.379 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:50.374+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:50.398 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:50.398 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:50.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:50.461 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:50.461 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:50.478 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:50.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:50.495 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:50.501 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:50.498+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:50.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:50.513 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:50.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:50.530 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:50.549 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:50.550 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:50.581 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:50.581+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:50.583 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:50.583 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:50.583 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:50.583 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:50.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:50.685 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:50.682+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:50.783 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:50.781+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:50.793 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:50.794 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:50.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:50.905 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:50.902+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:50.985 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:50.982+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:50.994 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.331928286999997, 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:50.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:28:50.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:50.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:50.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352686781999999, 2024-05-06T00:28:51.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.004 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352612563000001, 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.005 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:28:51.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:28:51.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.009 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352550663999999, 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:28:51.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352491065000002, 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:28:51.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:28:51.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352406930000001, 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:28:51.018 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:28:51.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:28:51.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352341416999998, 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:28:51.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:28:51.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352283555, 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.028 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352186959999997, 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.033 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:28:51.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:28:51.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352109235, 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:28:51.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.042 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.352036449000003, 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:28:51.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.351958957999997, 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.047 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:28:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:28:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:28:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.053 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.351864829, 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.054 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:28:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:28:51.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.060 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.351781867, 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:28:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.351707761999997, 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:28:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:28:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:28:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:28:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:28:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:28:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:28:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:28:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.351608573, 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:28:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:28:51.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.351545420999997, 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:28:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.351445155999997, 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:28:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.351355783999999, 2024-05-06T00:28:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:28:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.105497100999997, 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.097 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:51.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:51.099 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:51.094+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:51.199 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:51.194+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:51.288 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:51.288 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:51.289 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:51.289 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:51.299 INFO:teuthology.orchestra.run.smithi186.stdout:4774 punch from 0x970d600 to 0x970d800, (0x200 bytes) 2024-05-06T00:28:51.299 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:51.294+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:51.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:51.315 INFO:teuthology.orchestra.run.smithi186.stdout:4777 read 0xaffda00 thru 0xb00166b (0x3c6c bytes) 2024-05-06T00:28:51.316 INFO:teuthology.orchestra.run.smithi186.stdout:4779 read 0xa61c600 thru 0xa61c801 (0x202 bytes) 2024-05-06T00:28:51.317 INFO:teuthology.orchestra.run.smithi186.stdout:4783 read 0x9f3d600 thru 0x9f46889 (0x928a bytes) 2024-05-06T00:28:51.317 INFO:teuthology.orchestra.run.smithi186.stdout:4786 punch from 0xa5ef200 to 0xa5f8a00, (0x9800 bytes) 2024-05-06T00:28:51.324 INFO:teuthology.orchestra.run.smithi186.stdout:4787 read 0x4a9d800 thru 0x4aa5d1e (0x851f bytes) 2024-05-06T00:28:51.326 INFO:teuthology.orchestra.run.smithi186.stdout:4790 punch from 0x6c55800 to 0x6c5e000, (0x8800 bytes) 2024-05-06T00:28:51.333 INFO:teuthology.orchestra.run.smithi186.stdout:4791 write 0x5d3fa00 thru 0x5d4922f (0x9830 bytes) 2024-05-06T00:28:51.334 INFO:teuthology.orchestra.run.smithi186.stdout:4795 write 0x99e200 thru 0x9acea6 (0xeca7 bytes) 2024-05-06T00:28:51.336 INFO:teuthology.orchestra.run.smithi186.stdout:4796 read 0xd6200 thru 0xe408e (0xde8f bytes) 2024-05-06T00:28:51.391 INFO:teuthology.orchestra.run.smithi186.stdout:4801 trunc from 0xb177800 to 0x38e8000 2024-05-06T00:28:51.400 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:51.401+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:51.488 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:51.488 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:51.488 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:51.493 INFO:teuthology.orchestra.run.smithi186.stdout:4803 read 0x2e8f800 thru 0x2e9a25b (0xaa5c bytes) 2024-05-06T00:28:51.501 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:51.501+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:51.502 INFO:teuthology.orchestra.run.smithi186.stdout:4805 trunc from 0x38e8000 to 0x6857e00 2024-05-06T00:28:51.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:51.518 INFO:teuthology.orchestra.run.smithi186.stdout:4809 write 0x2989e00 thru 0x2996435 (0xc636 bytes) 2024-05-06T00:28:51.522 INFO:teuthology.orchestra.run.smithi186.stdout:4811 trunc from 0x6857e00 to 0x75f4200 2024-05-06T00:28:51.556 INFO:teuthology.orchestra.run.smithi186.stdout:4813 trunc from 0x75f4200 to 0xba2a000 2024-05-06T00:28:51.569 INFO:teuthology.orchestra.run.smithi186.stdout:4815 punch from 0x169a200 to 0x169e000, (0x3e00 bytes) 2024-05-06T00:28:51.577 INFO:teuthology.orchestra.run.smithi186.stdout:4819 trunc from 0xba2a000 to 0x8cd6000 2024-05-06T00:28:51.591 INFO:teuthology.orchestra.run.smithi186.stdout:4825 punch from 0x7d9f600 to 0x7da7400, (0x7e00 bytes) 2024-05-06T00:28:51.592 INFO:teuthology.orchestra.run.smithi186.stdout:4826 write 0x85b6c00 thru 0x85c2738 (0xbb39 bytes) 2024-05-06T00:28:51.592 INFO:teuthology.orchestra.run.smithi186.stdout:4828 punch from 0x75cb000 to 0x75cb400, (0x400 bytes) 2024-05-06T00:28:51.598 INFO:teuthology.orchestra.run.smithi186.stdout:4829 trunc from 0x8cd6000 to 0xbb30a00 2024-05-06T00:28:51.602 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:51.598+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:51.620 INFO:teuthology.orchestra.run.smithi186.stdout:4831 write 0x9709200 thru 0x9717b18 (0xe919 bytes) 2024-05-06T00:28:51.621 INFO:teuthology.orchestra.run.smithi186.stdout:4832 read 0x2f92200 thru 0x2f989b1 (0x67b2 bytes) 2024-05-06T00:28:51.622 INFO:teuthology.orchestra.run.smithi186.stdout:4833 punch from 0x45f4800 to 0x45f6c00, (0x2400 bytes) 2024-05-06T00:28:51.643 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:51.643 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.183956808000005, 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:28:51.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:28:51.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:28:51.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.648 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.183154818999995, 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:28:51.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:28:51.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.183068364999997, 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:28:51.653 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:28:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.182995771999998, 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:28:51.658 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:28:51.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:28:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:28:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:28:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:28:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.668 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.669 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.182935920999995, 2024-05-06T00:28:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:28:51.670 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:28:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:51.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:28:51.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:28:51.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:51.677 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.182860238000004, 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:28:51.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:28:51.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:28:51.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:28:51.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:28:51.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.175154664999994, 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:28:51.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:28:51.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.175100458000003, 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:28:51.688 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:28:51.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:28:51.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.175048309000005, 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.693 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:28:51.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:28:51.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.173090594000001, 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:28:51.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:28:51.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.173014253999995, 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.702 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:28:51.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:28:51.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.357876547000004, 2024-05-06T00:28:51.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:28:51.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:28:51.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:28:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:28:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:28:51.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.357824105999995, 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:28:51.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:28:51.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:28:51.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:28:51.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.719 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.350804123000003, 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:51.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:28:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.350757411999993, 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:28:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:28:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.350427303999993, 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:28:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:28:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.350399382000006, 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:28:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:28:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:28:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.339532492999993, 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.681882691999999, 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:28:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:28:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:28:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:28:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:28:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:28:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:28:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:28:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:28:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:28:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:28:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:28:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.680698076999999, 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:28:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:28:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:51.764 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:51.765 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:28:51.766 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:51.765+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:51.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:51.867 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:51.862+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:51.907 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:51.908 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:51.908 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:51.908 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:51.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:51.972 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:51.973+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:52.038 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:52.039 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:52.039 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:52.039 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:52.039 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:52.052 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:52.097 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:52.097+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:52.172 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:52.170+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:52.213 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 113.407838392, 2024-05-06T00:28:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:28:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:52.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:28:52.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:28:52.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:28:52.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:28:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:28:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 113.407822189, 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:28:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:28:52.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:28:52.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:28:52.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34113 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 845824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399692107, 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.073299384, 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34113 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2200000000000004e-07 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812509+0000", 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-06 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812519+0000", 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6979999999999993e-06 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812528+0000", 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5370000000000003e-06 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817889+0000", 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053609809999999999 2024-05-06T00:28:52.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817903+0000", 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3545e-05 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885659+0000", 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.067755797 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885734+0000", 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5698999999999995e-05 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885805+0000", 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0442000000000001e-05 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34114 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 851968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399660071, 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0734813590000001, 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34114 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.235 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:52.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-07 2024-05-06T00:28:52.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812540+0000", 2024-05-06T00:28:52.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4509999999999999e-06 2024-05-06T00:28:52.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:52.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5069999999999997e-06 2024-05-06T00:28:52.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812552+0000", 2024-05-06T00:28:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817936+0000", 2024-05-06T00:28:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053840650000000004 2024-05-06T00:28:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817944+0000", 2024-05-06T00:28:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5890000000000003e-06 2024-05-06T00:28:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885879+0000", 2024-05-06T00:28:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0679342119999999 2024-05-06T00:28:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885952+0000", 2024-05-06T00:28:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2924999999999997e-05 2024-05-06T00:28:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886019+0000", 2024-05-06T00:28:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7358000000000001e-05 2024-05-06T00:28:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34115 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 858112~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399624961999997, 2024-05-06T00:28:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0736428419999999, 2024-05-06T00:28:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34115 2024-05-06T00:28:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:52.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9700000000000002e-07 2024-05-06T00:28:52.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812575+0000", 2024-05-06T00:28:52.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.402e-06 2024-05-06T00:28:52.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:52.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-05-06T00:28:52.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812587+0000", 2024-05-06T00:28:52.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7979999999999996e-06 2024-05-06T00:28:52.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817971+0000", 2024-05-06T00:28:52.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053842719999999998 2024-05-06T00:28:52.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817979+0000", 2024-05-06T00:28:52.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4250000000000001e-06 2024-05-06T00:28:52.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886063+0000", 2024-05-06T00:28:52.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068084475 2024-05-06T00:28:52.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886129+0000", 2024-05-06T00:28:52.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6109e-05 2024-05-06T00:28:52.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886216+0000", 2024-05-06T00:28:52.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6312999999999998e-05 2024-05-06T00:28:52.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.255 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.256 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.256 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:52.256 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399591972000003, 2024-05-06T00:28:52.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:28:52.256 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.257 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.257 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.257 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.257 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.257 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:28:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:28:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:28:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:28:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:28:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:28:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:28:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:28:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:28:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:28:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:28:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:28:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:28:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399560602000001, 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:28:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:28:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:28:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399526813000001, 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:28:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:28:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:28:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:28:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399346356999999, 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:28:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:28:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:28:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399312713999997, 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:28:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:28:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:28:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:28:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:28:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399288931999997, 2024-05-06T00:28:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:28:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:28:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:28:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:28:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:28:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:28:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:28:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399262655000001, 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:28:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:28:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:28:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:28:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:28:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399219414000001, 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:28:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:28:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:28:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:28:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:28:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:28:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:28:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:28:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:28:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:28:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399189784999997, 2024-05-06T00:28:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:28:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399156935000001, 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:28:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:28:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:28:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399125296999998, 2024-05-06T00:28:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:28:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:28:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399092494999998, 2024-05-06T00:28:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:28:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399060462000001, 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.399028383000001, 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:52.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:28:52.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.398997074999997, 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:28:52.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:28:52.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:52.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:28:52.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:28:52.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.356 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.356 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.356 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:52.358 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:52.359 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:52.358+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:52.458 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:52.458+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:52.485 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:52.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:52.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3.80746148 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.86320389600000003, 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.86322768599999999, 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:28:52.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:28:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3.80746148 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.86313261200000002, 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.86320363200000005, 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:28:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:28:52.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3.80746148 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.86310097900000005, 2024-05-06T00:28:52.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.863210229, 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:28:52.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3.d981cd80 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.858882433, 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.85902937199999996, 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:28:52.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:52.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:28:52.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:28:52.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:52.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:52.503 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:28:52.503 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 4 2024-05-06T00:28:52.503 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:52.504 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:52.560 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:52.557+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:52.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:52.623 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:52.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:52.623 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:52.623 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:52.624 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:52.629 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:28:52.651 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:52.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:52.867 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:52.868 INFO:teuthology.orchestra.run.smithi186.stdout:4834 write 0xdfaf400 thru 0xdfb4953 (0x5554 bytes) 2024-05-06T00:28:52.868 INFO:teuthology.orchestra.run.smithi186.stdout:4835 trunc from 0xdfb4a00 to 0x7b14600 2024-05-06T00:28:52.868 INFO:teuthology.orchestra.run.smithi186.stdout:4836 read 0x5516000 thru 0x551e74a (0x874b bytes) 2024-05-06T00:28:52.868 INFO:teuthology.orchestra.run.smithi186.stdout:4839 write 0x1897000 thru 0x18a4391 (0xd392 bytes) 2024-05-06T00:28:52.868 INFO:teuthology.orchestra.run.smithi186.stdout:4840 punch from 0x6f30a00 to 0x6f31600, (0xc00 bytes) 2024-05-06T00:28:52.868 INFO:teuthology.orchestra.run.smithi186.stdout:4845 read 0x50f400 thru 0x51c178 (0xcd79 bytes) 2024-05-06T00:28:52.868 INFO:teuthology.orchestra.run.smithi186.stdout:4846 write 0xe920000 thru 0xe926d61 (0x6d62 bytes) 2024-05-06T00:28:52.869 INFO:teuthology.orchestra.run.smithi186.stdout:4847 punch from 0x986f600 to 0x9873400, (0x3e00 bytes) 2024-05-06T00:28:52.869 INFO:teuthology.orchestra.run.smithi186.stdout:4848 punch from 0x1981a00 to 0x198fc00, (0xe200 bytes) 2024-05-06T00:28:52.869 INFO:teuthology.orchestra.run.smithi186.stdout:4849 trunc from 0xe926e00 to 0x922e00 2024-05-06T00:28:52.869 INFO:teuthology.orchestra.run.smithi186.stdout:4850 punch from 0x8f5a00 to 0x901200, (0xb800 bytes) 2024-05-06T00:28:52.871 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:52.866+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:52.892 INFO:teuthology.orchestra.run.smithi186.stdout:4852 write 0x882ae00 thru 0x8834340 (0x9541 bytes) 2024-05-06T00:28:52.895 INFO:teuthology.orchestra.run.smithi186.stdout:4855 write 0x698a600 thru 0x698dd9f (0x37a0 bytes) 2024-05-06T00:28:52.896 INFO:teuthology.orchestra.run.smithi186.stdout:4857 read 0x2367000 thru 0x23683bf (0x13c0 bytes) 2024-05-06T00:28:52.897 INFO:teuthology.orchestra.run.smithi186.stdout:4860 write 0x16d4200 thru 0x16de0aa (0x9eab bytes) 2024-05-06T00:28:52.936 INFO:teuthology.orchestra.run.smithi186.stdout:4862 write 0xab99c00 thru 0xaba0f44 (0x7345 bytes) 2024-05-06T00:28:52.938 INFO:teuthology.orchestra.run.smithi186.stdout:4864 clone 32 order 22 su 2097152 sc 4 2024-05-06T00:28:52.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:52.944 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:52.967 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:52.965+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:52.968 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:52.968 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:52.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:53.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:53.035 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:53.035 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:53.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:53.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:53.105 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:53.105+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:53.169 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:53.169+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:53.223 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:53.223 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:53.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:53.281 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:53.281+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:53.371 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:53.369+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:53.404 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:53.404 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:53.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:53.472 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:53.470+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:53.573 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:53.573+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:53.584 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:53.584 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:53.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:53.674 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:53.673+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:53.731 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:53.731 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:53.731 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-05-06T00:28:53.731 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-05-06T00:28:53.737 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:53.737 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:53.752 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:53.776 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:53.777+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:53.814 INFO:teuthology.orchestra.run.smithi186.stdout:flattening image image_client.0-clone31 2024-05-06T00:28:53.876 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:53.877+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:53.878 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:53.878 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:53.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:53.977 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:53.977+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:54.010 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:54.010 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:54.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:54.052 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-06T00:28:54.049+0000 7fdfb36a9640 -1 mon.a@0(leader).osd e122 definitely_dead 0 2024-05-06T00:28:54.077 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:54.077+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:54.151 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:54.151 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:54.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:54.179 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:54.177+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:54.280 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:54.274+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:54.290 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:54.290 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:54.304 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:54.393 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:54.390+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:54.437 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-05-06T00:28:54.472 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:54.473 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:54.481 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:54.478+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:54.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:54.531 INFO:teuthology.orchestra.run.smithi186.stdout:4867 punch from 0x8e1d600 to 0x8e28000, (0xaa00 bytes) 2024-05-06T00:28:54.582 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:54.578+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:54.602 INFO:teuthology.orchestra.run.smithi186.stdout:4868 trunc from 0xaba1000 to 0x689e400 2024-05-06T00:28:54.628 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.2. 2024-05-06T00:28:54.657 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-05-06T00:28:54.669 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:54.669 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:54.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:54.683 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:54.678+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:54.783 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:54.785+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:54.844 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:54.844 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:54.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:54.867 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:28:54.905 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:54.902+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:54.981 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:54.981 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:54.985 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:54.982+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:54.995 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:55.063 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:55.086 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:55.086+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:55.111 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:55.111 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:55.126 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:55.187 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:55.182+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:55.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:55.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:55.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:55.242 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:55.242 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:55.242 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:55.242 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:55.242 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:55.255 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:55.256 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:55.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:55.277 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:55.287 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:55.286+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:55.297 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:55.370 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:55.370 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:55.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:55.370 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:55.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.370 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:55.370 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:55.370 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.56625347799999, 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:55.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.56623727500001, 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.376 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:28:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34113 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 845824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.558107192999998, 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.073299384, 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34113 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812506+0000", 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2200000000000004e-07 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812509+0000", 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.163e-06 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812519+0000", 2024-05-06T00:28:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6979999999999993e-06 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812528+0000", 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5370000000000003e-06 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817889+0000", 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053609809999999999 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817903+0000", 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3545e-05 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885659+0000", 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.067755797 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885734+0000", 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5698999999999995e-05 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885805+0000", 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0442000000000001e-05 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34114 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 851968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.558075156999998, 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0734813590000001, 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34114 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812538+0000", 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6800000000000001e-07 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812540+0000", 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4509999999999999e-06 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812547+0000", 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5069999999999997e-06 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812552+0000", 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7840000000000003e-06 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817936+0000", 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053840650000000004 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817944+0000", 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5890000000000003e-06 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885879+0000", 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0679342119999999 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.885952+0000", 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2924999999999997e-05 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886019+0000", 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7358000000000001e-05 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34115 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 858112~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.558040048000002, 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0736428419999999, 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34115 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812573+0000", 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9700000000000002e-07 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812575+0000", 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.402e-06 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812582+0000", 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812587+0000", 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7979999999999996e-06 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817971+0000", 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053842719999999998 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.817979+0000", 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4250000000000001e-06 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886063+0000", 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068084475 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886129+0000", 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6109e-05 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886216+0000", 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6312999999999998e-05 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.558007058000001, 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557975687999999, 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:55.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:28:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:28:55.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557941898999999, 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.407 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:28:55.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:28:55.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557761442999997, 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.412 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:28:55.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:28:55.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557727800000002, 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:28:55.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:28:55.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557704018000003, 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.423 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:28:55.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:28:55.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:55.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:28:55.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557677740999999, 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:28:55.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:28:55.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557634499999999, 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:28:55.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557604871000002, 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:28:55.441 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:28:55.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:28:55.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:28:55.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557572020999999, 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:28:55.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:28:55.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:28:55.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557540383000003, 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:28:55.451 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:28:55.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557507581000003, 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:28:55.456 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.461 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557475547999999, 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:28:55.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:28:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557443468999999, 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.471 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.557412161000002, 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.472 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.473 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.473 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.473 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.473 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:28:55.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.473 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:28:55.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:28:55.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.477 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:55.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:55.479 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:55.477+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:55.579 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:55.577+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:55.645 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:28:55.650 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:55.650 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:55.670 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:55.717 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:55.717+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:55.781 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:55.782+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:55.829 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:55.829 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:55.830 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:55.845 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:55.888 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:55.889+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:55.963 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.963 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936493626000001, 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:28:55.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936472836999997, 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936452774999999, 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.976 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936433499000003, 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:28:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:28:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936413381000001, 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:28:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:28:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936381972, 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:55.989 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936349059999998, 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936328473000003, 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:28:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:55.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936309092999998, 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:28:55.998 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:55.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:28:56.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936264412, 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:28:56.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:28:56.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:28:56.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936199320999997, 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:28:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:28:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936167746999999, 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936145486000001, 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:28:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:28:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.936124370000002, 2024-05-06T00:28:56.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.278407743999999, 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:28:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.277997501999998, 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:28:56.033 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3393529080000004, 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:28:56.038 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3392816239999998, 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:28:56.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:28:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.339249991, 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:28:56.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:28:56.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:28:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3350314450000003, 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:28:56.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:28:56.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:28:56.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.055 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:56.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:56.057 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:56.057+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:56.158 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:56.154+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:56.225 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:28:56.226 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:56.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:56.259 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:56.258+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:56.354 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:56.354 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:28:56.354 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:28:56.354 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:28:56.355 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:56.360 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:56.361+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:56.368 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:56.461 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:56.462+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:56.538 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:28:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:28:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:28:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.530598802, 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:28:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:28:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.530370230000003, 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:28:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:28:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.530292946000003, 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:28:56.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:28:56.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.530096964999998, 2024-05-06T00:28:56.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.622987588999997, 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:28:56.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.622926710000002, 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:28:56.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:56.565 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.505370259999999, 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:28:56.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.505325532000001, 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:28:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:28:56.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.505287131000003, 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:28:56.577 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:28:56.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:28:56.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.505249868999996, 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:28:56.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:28:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.505212960000001, 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:28:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:28:56.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.505175041999998, 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:28:56.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:28:56.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.505137927, 2024-05-06T00:28:56.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:28:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.499357478, 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:28:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.499319393999997, 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:56.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:56.608 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.499281719999999, 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.489846210000003, 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:28:56.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.616 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.489813783000002, 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:28:56.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.489789901000002, 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.621 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:28:56.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:28:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.489769971000001, 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:28:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:28:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:28:56.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:28:56.628 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:28:56.629 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:28:56.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:56.631 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:56.630+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:56.731 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:56.726+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:56.745 INFO:teuthology.orchestra.run.smithi186.stdout:4870 punch from 0x32f0a00 to 0x32fe800, (0xde00 bytes) 2024-05-06T00:28:56.748 INFO:teuthology.orchestra.run.smithi186.stdout:4872 read 0x42b4200 thru 0x42be727 (0xa528 bytes) 2024-05-06T00:28:56.748 INFO:teuthology.orchestra.run.smithi186.stdout:4880 read 0x4d16000 thru 0x4d18487 (0x2488 bytes) 2024-05-06T00:28:56.749 INFO:teuthology.orchestra.run.smithi186.stdout:4881 clone 33 order 18 su 16384 sc 12 2024-05-06T00:28:56.766 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:56.766 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:56.767 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:56.767 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:56.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:56.869 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:56.866+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:56.932 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:56.933+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:56.946 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:56.946 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:56.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:28:57.034 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:57.033+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:57.126 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 147.52340918900001, 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:28:57.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:28:57.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.133 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250760490000005, 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:28:57.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:28:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:28:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250725313999993, 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:28:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:28:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:28:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250636635000006, 2024-05-06T00:28:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:28:57.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:28:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:28:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:28:57.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250534692000002, 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:28:57.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:28:57.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.159 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250425590000006, 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:28:57.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.165 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250327730999999, 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:28:57.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:28:57.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:28:57.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:28:57.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.172 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250262680999995, 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:28:57.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:28:57.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250170034999996, 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.179 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:28:57.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:28:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 99.250098030999993, 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:28:57.185 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:28:57.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:28:57.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:28:57.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:28:57.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:57.192 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490636545000001, 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:57.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:28:57.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490617297, 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.198 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:28:57.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490567177000003, 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.204 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.209 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490546125999998, 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:28:57.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.215 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490530806999999, 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:57.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490513925000002, 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.221 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:28:57.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.226 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490495589000005, 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.227 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:28:57.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:28:57.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490479586999996, 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:28:57.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:28:57.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:28:57.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:28:57.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.490393224000002, 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:28:57.238 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:28:57.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.480430748000003, 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:28:57.244 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:28:57.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:28:57.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:28:57.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:28:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:28:57.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:28:57.250 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:57.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:28:57.252 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:57.249+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:57.315 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:28:57.360 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:57.361+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:57.381 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:57.381 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:57.381 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:57.381 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:57.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:28:57.453 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:57.453+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:57.476 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:57.477 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:57.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:57.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:57.510 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:57.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:57.530 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:57.532 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:57.532 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:57.532 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:28:57.532 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:28:57.532 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:57.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:28:57.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:57.548 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:57.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:57.568 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:57.593 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:57.590+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:57.655 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:57.653+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:57.673 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:28:57.674 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:57.674 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:57.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:28:57.777 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:57.778+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:57.790 INFO:teuthology.orchestra.run.smithi186.stdout:leaving image image_client.0-clone32 intact 2024-05-06T00:28:57.857 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:57.857+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:57.862 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:57.862 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:57.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:28:57.965 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:57.966+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:58.045 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:58.045 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:58.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:28:58.060 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:58.057+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:58.160 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:58.161+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:58.227 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:58.227 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:58.242 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:28:58.261 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:58.262+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:58.362 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:58.362+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:58.372 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:58.372 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:58.386 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:28:58.463 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:58.461+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:58.509 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:58.509 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:58.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:28:58.564 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:58.562+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:58.658 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:28:58.666 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:58.665+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:58.696 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:58.696 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:58.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:28:58.796 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:58.797+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:58.867 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:58.866+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:58.873 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:58.874 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:58.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:28:58.968 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:58.966+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:59.054 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:59.054 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:59.069 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:28:59.069 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:59.069+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:28:59.170 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:59.170+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:59.232 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:59.233 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:59.248 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:28:59.292 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:28:59.293+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:28:59.365 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:59.365 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:59.372 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:59.370+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:59.379 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:28:59.473 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:59.474+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:59.545 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:59.546 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:59.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:28:59.586 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:28:59.608 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:28:59.609+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:28:59.658 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:28:59.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:28:59.658 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-05-06T00:28:59.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-05-06T00:28:59.675 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:28:59.674+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:28:59.706 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:28:59.706 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:28:59.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:28:59.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:28:59.758 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:28:59.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:28:59.774 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:28:59.777 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:28:59.774+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:28:59.791 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:28:59.791 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:28:59.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:28:59.810 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:28:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:28:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:28:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:28:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:28:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:28:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:28:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:28:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:28:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:28:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:28:59.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:28:59.832 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:28:59.832 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:28:59.832 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:28:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:28:59.834 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:28:59.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-05-06T00:28:59.852 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:28:59.852 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:28:59.864 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:28:59.864 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:28:59.864 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:28:59.864 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:28:59.877 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:28:59.878+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:28:59.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:28:59.977 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:28:59.977+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:00.002 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:00.002 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:00.002 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:00.002 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:00.002 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:00.002 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:00.004 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:00.005 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-05-06T00:29:00.029 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-05-06T00:29:00.047 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:00.047 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:00.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:00.109 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:00.110+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:00.179 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:00.177+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:00.184 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.184 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 150.58123290399999, 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:29:00.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:29:00.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:29:00.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:00.191 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.308584205, 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:29:00.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:29:00.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:29:00.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.30854902900001, 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.197 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:29:00.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:29:00.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.203 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.30846035, 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:29:00.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:29:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:29:00.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:29:00.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.308358407, 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.210 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:29:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:29:00.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:29:00.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:29:00.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:29:00.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:29:00.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:29:00.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.308249305, 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.218 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:29:00.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.308151446, 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.224 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:29:00.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:29:00.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.30808639599999, 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.230 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.30799374999999, 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:29:00.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:29:00.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.30792174600001, 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.242 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:29:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:29:00.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:29:00.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.54846026, 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:29:00.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:29:00.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:29:00.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.548441012, 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:29:00.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:29:00.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.548390892, 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:29:00.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.267 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.548369841, 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.268 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:29:00.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:29:00.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.548354522, 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.274 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:29:00.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.54833764, 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:29:00.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:29:00.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:29:00.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.548319304, 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:29:00.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:00.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.54830330199999, 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:29:00.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:29:00.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:29:00.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.548216939, 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:29:00.297 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:29:00.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:00.303 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.538254463, 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:00.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.309 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:00.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:00.312 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:00.310+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:00.313 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:00.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-05-06T00:29:00.411 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:00.410+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:00.493 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:00.493 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:00.494 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:00.494 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:00.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:00.525 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:00.525 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:00.525 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:00.525 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:00.526 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:00.526 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:00.526 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:00.526 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:00.526 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:00.526 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:00.526 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:00.526 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:00.529 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:00.529 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-05-06T00:29:00.553 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:00.554+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:00.613 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:00.613+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:00.682 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:00.683 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:00.683 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:00.683 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:00.683 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:00.691 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:29:00.693 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:00.693 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:00.693 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:00.693 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:00.697 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:00.697 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-05-06T00:29:00.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:00.745 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:00.746+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:00.816 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:00.814+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:00.825 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.165435783000007, 2024-05-06T00:29:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.186194278000002, 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.186120058999997, 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.186058160000002, 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185998560999998, 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185914425999997, 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185848913000001, 2024-05-06T00:29:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185791051000002, 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185694456, 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185616731000003, 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185543944999999, 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185466454, 2024-05-06T00:29:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185372325000003, 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185289363000003, 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185215257999999, 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185116069000003, 2024-05-06T00:29:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.185052917, 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.184952652, 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.184863280000002, 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.939004597, 2024-05-06T00:29:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:00.924 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:00.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:00.928 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:00.928 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:00.925+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:01.026 INFO:teuthology.orchestra.run.smithi186.stdout:4882 write 0x4cf3400 thru 0x4d01b68 (0xe769 bytes) 2024-05-06T00:29:01.026 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:01.026+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:01.046 INFO:teuthology.orchestra.run.smithi186.stdout:4884 trunc from 0x689e400 to 0xa2ef400 2024-05-06T00:29:01.105 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:01.105 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:01.110 INFO:teuthology.orchestra.run.smithi186.stdout:4885 trunc from 0xa2ef400 to 0x5e22000 2024-05-06T00:29:01.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:01.126 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:01.125+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:01.173 INFO:teuthology.orchestra.run.smithi186.stdout:4887 write 0xa41a400 thru 0xa4257fb (0xb3fc bytes) 2024-05-06T00:29:01.174 INFO:teuthology.orchestra.run.smithi186.stdout:4890 write 0x14c1000 thru 0x14c7302 (0x6303 bytes) 2024-05-06T00:29:01.175 INFO:teuthology.orchestra.run.smithi186.stdout:4891 read 0x3969c00 thru 0x39719a9 (0x7daa bytes) 2024-05-06T00:29:01.226 INFO:teuthology.orchestra.run.smithi186.stdout:4893 write 0xc90c200 thru 0xc91924f (0xd050 bytes) 2024-05-06T00:29:01.227 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:01.225+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:01.260 INFO:teuthology.orchestra.run.smithi186.stdout:4895 write 0xd08b000 thru 0xd099f79 (0xef7a bytes) 2024-05-06T00:29:01.261 INFO:teuthology.orchestra.run.smithi186.stdout:4896 read 0xbe2e600 thru 0xbe383ba (0x9dbb bytes) 2024-05-06T00:29:01.262 INFO:teuthology.orchestra.run.smithi186.stdout:4897 write 0x89a6c00 thru 0x89b00fe (0x94ff bytes) 2024-05-06T00:29:01.262 INFO:teuthology.orchestra.run.smithi186.stdout:4898 write 0x930e000 thru 0x93195f9 (0xb5fa bytes) 2024-05-06T00:29:01.273 INFO:teuthology.orchestra.run.smithi186.stdout:4900 trunc from 0xd09a000 to 0x870ac00 2024-05-06T00:29:01.300 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:01.301 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:01.301 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:01.301 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:01.301 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:01.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:01.336 INFO:teuthology.orchestra.run.smithi186.stdout:4902 write 0x4ff000 thru 0x50a035 (0xb036 bytes) 2024-05-06T00:29:01.338 INFO:teuthology.orchestra.run.smithi186.stdout:4906 punch from 0x4f02800 to 0x4f08400, (0x5c00 bytes) 2024-05-06T00:29:01.341 INFO:teuthology.orchestra.run.smithi186.stdout:4907 punch from 0x18e4400 to 0x18e5800, (0x1400 bytes) 2024-05-06T00:29:01.367 INFO:teuthology.orchestra.run.smithi186.stdout:4910 write 0xb6d3c00 thru 0xb6dc3c6 (0x87c7 bytes) 2024-05-06T00:29:01.368 INFO:teuthology.orchestra.run.smithi186.stdout:4911 punch from 0x4aa7200 to 0x4aaba00, (0x4800 bytes) 2024-05-06T00:29:01.369 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:01.370+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:01.369 INFO:teuthology.orchestra.run.smithi186.stdout:4912 read 0xfb6400 thru 0xfc44c8 (0xe0c9 bytes) 2024-05-06T00:29:01.370 INFO:teuthology.orchestra.run.smithi186.stdout:4914 write 0x5472800 thru 0x5477613 (0x4e14 bytes) 2024-05-06T00:29:01.370 INFO:teuthology.orchestra.run.smithi186.stdout:4915 read 0xa617200 thru 0xa623e92 (0xcc93 bytes) 2024-05-06T00:29:01.371 INFO:teuthology.orchestra.run.smithi186.stdout:4916 write 0xb1c6200 thru 0xb1c77b1 (0x15b2 bytes) 2024-05-06T00:29:01.371 INFO:teuthology.orchestra.run.smithi186.stdout:4919 read 0xca8e00 thru 0xcae385 (0x5586 bytes) 2024-05-06T00:29:01.372 INFO:teuthology.orchestra.run.smithi186.stdout:4920 punch from 0xb562c00 to 0xb563c00, (0x1000 bytes) 2024-05-06T00:29:01.381 INFO:teuthology.orchestra.run.smithi186.stdout:4921 trunc from 0xb6dc400 to 0xd49d000 2024-05-06T00:29:01.393 INFO:teuthology.orchestra.run.smithi186.stdout:4923 punch from 0x1857600 to 0x1861a00, (0xa400 bytes) 2024-05-06T00:29:01.423 INFO:teuthology.orchestra.run.smithi186.stdout:4926 trunc from 0xd49d000 to 0x38b1400 2024-05-06T00:29:01.429 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:01.430+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:01.477 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:01.477 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:01.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:01.477 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:01.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.477 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.01332001, 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:29:01.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2861 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 86016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.01251802100001, 2024-05-06T00:29:01.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375365000000001, 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2861 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459978+0000", 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459980+0000", 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459987+0000", 2024-05-06T00:29:01.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2030000000000003e-06 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460010+0000", 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2959000000000002e-05 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460026+0000", 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486086+0000", 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026059718999999999 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486186+0000", 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010087299999999999 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486354+0000", 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016732100000000001 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.01243156699999, 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:29:01.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.01235897399999, 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:29:01.491 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:29:01.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.01229912300001, 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:29:01.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:29:01.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.01222344, 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.500 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:29:01.504 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.004517867, 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:29:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.00446366, 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:29:01.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:01.513 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.004411511, 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:29:01.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:29:01.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3041 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.002453796, 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026427428999999999, 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.518 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3041 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470043+0000", 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470044+0000", 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2550000000000001e-06 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470050+0000", 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470069+0000", 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8904999999999999e-05 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470098+0000", 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.887e-05 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496314+0000", 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026216247000000002 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496397+0000", 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2552000000000005e-05 2024-05-06T00:29:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496470+0000", 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3232000000000006e-05 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.002377456, 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:29:01.522 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:29:01.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2892 3.13 3:cc6acb60:::rbd_data.110cd5ddb45c.0000000000000082:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.187239749, 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026482469000000002, 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2892 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285257+0000", 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.03e-07 2024-05-06T00:29:01.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285260+0000", 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285266+0000", 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285271+0000", 2024-05-06T00:29:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7849999999999999e-06 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.306984+0000", 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021713672999999999 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307049+0000", 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4900000000000005e-05 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:29:01.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307208+0000", 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015879199999999999 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311128+0000", 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039201519999999997 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311676+0000", 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054762599999999995 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:01.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311714+0000", 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7802000000000002e-05 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311739+0000", 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5624000000000001e-05 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2894 3.13 3:cc822c50:::rbd_data.110cd5ddb45c.0000000000000089:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:29:01.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.187187308000006, 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026483573999999999, 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2894 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:29:01.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285309+0000", 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285310+0000", 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285313+0000", 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2590000000000001e-06 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285319+0000", 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.178e-06 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.285325+0000", 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0100000000000001e-06 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307634+0000", 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022309091999999999 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307697+0000", 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2702999999999999e-05 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.307888+0000", 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099399999999999 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311178+0000", 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032894339999999999 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311747+0000", 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000569251 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311775+0000", 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8558e-05 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.311793+0000", 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7285000000000001e-05 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.180167324999999, 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:29:01.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:29:01.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:29:01.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.180120614000003, 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:29:01.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.179790506000003, 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:29:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:29:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:29:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.179762584000002, 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:29:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:29:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.168895695000003, 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:01.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:29:01.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:29:01.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:29:01.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:29:01.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.511245893999998, 2024-05-06T00:29:01.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:01.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:01.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.510061278999999, 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:01.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:01.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:01.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:01.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:01.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:01.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:01.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:01.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:01.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.578 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:01.578 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:01.578 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:01.580 INFO:teuthology.orchestra.run.smithi186.stdout:4927 read 0x2b13000 thru 0x2b1db85 (0xab86 bytes) 2024-05-06T00:29:01.580 INFO:teuthology.orchestra.run.smithi186.stdout:4928 write 0xa234600 thru 0xa2408cd (0xc2ce bytes) 2024-05-06T00:29:01.580 INFO:teuthology.orchestra.run.smithi186.stdout:4930 read 0x2589400 thru 0x2595947 (0xc548 bytes) 2024-05-06T00:29:01.580 INFO:teuthology.orchestra.run.smithi186.stdout:4932 write 0x71e5800 thru 0x71f10b0 (0xb8b1 bytes) 2024-05-06T00:29:01.581 INFO:teuthology.orchestra.run.smithi186.stdout:4933 punch from 0x6e8b800 to 0x6e95400, (0x9c00 bytes) 2024-05-06T00:29:01.581 INFO:teuthology.orchestra.run.smithi186.stdout:4935 trunc from 0xa240a00 to 0x194ba00 2024-05-06T00:29:01.581 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:01.578+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:01.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:01.600 INFO:teuthology.orchestra.run.smithi186.stdout:4938 punch from 0x349000 to 0x358400, (0xf400 bytes) 2024-05-06T00:29:01.623 INFO:teuthology.orchestra.run.smithi186.stdout:4939 write 0x3411600 thru 0x3418982 (0x7383 bytes) 2024-05-06T00:29:01.646 INFO:teuthology.orchestra.run.smithi186.stdout:4941 write 0x49ec600 thru 0x49eda0a (0x140b bytes) 2024-05-06T00:29:01.665 INFO:teuthology.orchestra.run.smithi186.stdout:4942 write 0x5552e00 thru 0x55549a4 (0x1ba5 bytes) 2024-05-06T00:29:01.680 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:01.678+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:01.691 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:29:01.709 INFO:teuthology.orchestra.run.smithi186.stdout:4943 write 0xb70a800 thru 0xb71319f (0x89a0 bytes) 2024-05-06T00:29:01.710 INFO:teuthology.orchestra.run.smithi186.stdout:4944 read 0x9853c00 thru 0x9854fa4 (0x13a5 bytes) 2024-05-06T00:29:01.710 INFO:teuthology.orchestra.run.smithi186.stdout:4945 write 0xa8c6000 thru 0xa8c91f4 (0x31f5 bytes) 2024-05-06T00:29:01.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:01.729 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:01.729 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:01.729 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:01.733 INFO:teuthology.orchestra.run.smithi186.stdout:4946 trunc from 0xb713200 to 0x1f41800 2024-05-06T00:29:01.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:01.784 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:01.785+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:01.823 INFO:teuthology.orchestra.run.smithi186.stdout:4947 write 0x1eba00 thru 0x1f4f14 (0x9515 bytes) 2024-05-06T00:29:01.827 INFO:teuthology.orchestra.run.smithi186.stdout:4948 punch from 0x8dfe00 to 0x8ed000, (0xd200 bytes) 2024-05-06T00:29:01.832 INFO:teuthology.orchestra.run.smithi186.stdout:4949 read 0x1f22e00 thru 0x1f2638f (0x3590 bytes) 2024-05-06T00:29:01.870 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:01.878 INFO:teuthology.orchestra.run.smithi186.stdout:4951 trunc from 0x1f41800 to 0xe936e00 2024-05-06T00:29:01.905 INFO:teuthology.orchestra.run.smithi186.stdout:4954 write 0x3726c00 thru 0x3730d49 (0xa14a bytes) 2024-05-06T00:29:01.905 INFO:teuthology.orchestra.run.smithi186.stdout:4955 read 0x7fac200 thru 0x7fbab42 (0xe943 bytes) 2024-05-06T00:29:01.906 INFO:teuthology.orchestra.run.smithi186.stdout:4956 write 0xae9be00 thru 0xaea9e42 (0xe043 bytes) 2024-05-06T00:29:01.906 INFO:teuthology.orchestra.run.smithi186.stdout:4959 read 0xe5ab400 thru 0xe5ae1ab (0x2dac bytes) 2024-05-06T00:29:01.909 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:01.909 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:01.910 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:01.910 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:01.909+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:01.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:01.953 INFO:teuthology.orchestra.run.smithi186.stdout:4960 trunc from 0xe936e00 to 0x5cf600 2024-05-06T00:29:01.984 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:01.982+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:02.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:02.034 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:02.034 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:02.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:02.052 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:02.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:02.071 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:02.084 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:02.085+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:02.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:02.090 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:02.100 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:02.101 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:02.113 INFO:teuthology.orchestra.run.smithi186.stdout:4962 write 0x9864000 thru 0x986ae44 (0x6e45 bytes) 2024-05-06T00:29:02.114 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:02.115 INFO:teuthology.orchestra.run.smithi186.stdout:4963 write 0xa86000 thru 0xa8f8e3 (0x98e4 bytes) 2024-05-06T00:29:02.115 INFO:teuthology.orchestra.run.smithi186.stdout:4964 read 0x19dee00 thru 0x19df0af (0x2b0 bytes) 2024-05-06T00:29:02.116 INFO:teuthology.orchestra.run.smithi186.stdout:4966 read 0x66c2a00 thru 0x66c8240 (0x5841 bytes) 2024-05-06T00:29:02.116 INFO:teuthology.orchestra.run.smithi186.stdout:4967 read 0x4df8200 thru 0x4dfadf8 (0x2bf9 bytes) 2024-05-06T00:29:02.116 INFO:teuthology.orchestra.run.smithi186.stdout:4969 read 0x82d9200 thru 0x82da589 (0x138a bytes) 2024-05-06T00:29:02.116 INFO:teuthology.orchestra.run.smithi186.stdout:4970 write 0x4f5f400 thru 0x4f65b87 (0x6788 bytes) 2024-05-06T00:29:02.116 INFO:teuthology.orchestra.run.smithi186.stdout:4975 read 0x4833800 thru 0x4842628 (0xee29 bytes) 2024-05-06T00:29:02.116 INFO:teuthology.orchestra.run.smithi186.stdout:4977 read 0x638a000 thru 0x6393a35 (0x9a36 bytes) 2024-05-06T00:29:02.116 INFO:teuthology.orchestra.run.smithi186.stdout:4978 read 0x141a00 thru 0x14dc3f (0xc240 bytes) 2024-05-06T00:29:02.118 INFO:teuthology.orchestra.run.smithi186.stdout:4979 punch from 0x735ce00 to 0x7368a00, (0xbc00 bytes) 2024-05-06T00:29:02.119 INFO:teuthology.orchestra.run.smithi186.stdout:4985 write 0x79adc00 thru 0x79adc1c (0x1d bytes) 2024-05-06T00:29:02.119 INFO:teuthology.orchestra.run.smithi186.stdout:4986 clone 34 order 23 su 32768 sc 3 2024-05-06T00:29:02.204 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:02.205+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:02.280 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:02.280 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:02.287 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:02.286+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:02.293 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:02.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:02.314 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:02.387 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:02.386+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:02.434 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:02.434 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:02.448 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:02.492 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:02.493+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:02.568 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:02.568 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:02.582 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:02.589 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:02.590+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:02.690 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:02.689+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:02.748 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:02.748 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:02.762 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:02.804 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:02.805+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:02.880 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone33 from 0x986b000 (overlap 0x5cf600) to 0x129600 2024-05-06T00:29:02.894 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:02.894+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:02.939 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:02.940 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:02.955 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:03.000 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:03.001+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:03.074 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:03.075 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:03.088 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:03.133 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:03.134+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:03.194 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:03.194+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:03.224 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:03.224 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:03.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:03.325 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:03.326+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:03.396 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:03.393+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:03.409 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:03.409 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:03.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:03.479 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-05-06T00:29:03.497 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:03.497+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:03.594 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:03.594 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:03.598 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:03.598+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:03.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:03.701 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:03.702+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:03.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:29:03.788 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:03.788 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:03.800 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:03.797+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:03.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:03.901 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:03.901+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:03.976 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:03.976 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:03.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:04.037 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:04.038+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:04.103 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:04.102+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:04.122 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:04.122 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:04.137 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:04.203 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:04.201+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:04.304 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:04.304 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:04.305 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:04.306+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:04.318 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:04.332 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:04.356 INFO:teuthology.orchestra.run.smithi186.stdout:4987 write 0x342e400 thru 0x343b37b (0xcf7c bytes) 2024-05-06T00:29:04.358 INFO:teuthology.orchestra.run.smithi186.stdout:4989 read 0x266ea00 thru 0x267d2fb (0xe8fc bytes) 2024-05-06T00:29:04.403 INFO:teuthology.orchestra.run.smithi186.stdout:4991 write 0xc627600 thru 0xc6349df (0xd3e0 bytes) 2024-05-06T00:29:04.405 INFO:teuthology.orchestra.run.smithi186.stdout:4992 read 0x2cc0000 thru 0x2cc487e (0x487f bytes) 2024-05-06T00:29:04.405 INFO:teuthology.orchestra.run.smithi186.stdout:4995 write 0xdf7200 thru 0xdf8740 (0x1541 bytes) 2024-05-06T00:29:04.405 INFO:teuthology.orchestra.run.smithi186.stdout:4996 read 0xc0dfa00 thru 0xc0e5f61 (0x6562 bytes) 2024-05-06T00:29:04.406 INFO:teuthology.orchestra.run.smithi186.stdout:4997 read 0xbfc9800 thru 0xbfd4d12 (0xb513 bytes) 2024-05-06T00:29:04.406 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:04.405+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:04.406 INFO:teuthology.orchestra.run.smithi186.stdout:4998 write 0x4ca9e00 thru 0x4cb4c9f (0xaea0 bytes) 2024-05-06T00:29:04.406 INFO:teuthology.orchestra.run.smithi186.stdout:4999 clone 35 order 21 su 524288 sc 7 2024-05-06T00:29:04.490 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:04.490 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:04.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:04.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:04.507 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:04.506+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:04.510 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:04.511 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:04.528 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:04.528 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:04.546 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:04.562 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:04.563 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:04.580 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:04.580 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:04.623 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.16372036200001, 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:29:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:29:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2862 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 118784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.162831919, 2024-05-06T00:29:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026811746000000001, 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2862 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460065+0000", 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460066+0000", 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.266e-06 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460083+0000", 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7243000000000001e-05 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460102+0000", 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8122e-05 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460119+0000", 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7096e-05 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486445+0000", 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026326868 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486681+0000", 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000235288 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486877+0000", 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195863 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.162759326, 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:29:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.162699475, 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:29:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:29:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.16262379200001, 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:29:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2968 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 192512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.154918219, 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026614367999999999, 2024-05-06T00:29:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2968 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467979+0000", 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467980+0000", 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319999999999999e-06 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.467986+0000", 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.677e-06 2024-05-06T00:29:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468002+0000", 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6642000000000001e-05 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468017+0000", 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4848000000000001e-05 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494228+0000", 2024-05-06T00:29:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026210753999999999 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494356+0000", 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012820299999999999 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494593+0000", 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023691200000000001 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2969 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 225280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.154864012, 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026789492000000002, 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2969 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468033+0000", 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468034+0000", 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5680000000000001e-06 2024-05-06T00:29:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468039+0000", 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.006e-06 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468055+0000", 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5988e-05 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468068+0000", 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2865e-05 2024-05-06T00:29:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494658+0000", 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026589511999999999 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494754+0000", 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5801e-05 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494822+0000", 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8751999999999995e-05 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.15481186300001, 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:29:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3042 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.152777808, 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026710441000000001, 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470119+0000", 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470120+0000", 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3939999999999999e-06 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470126+0000", 2024-05-06T00:29:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8429999999999998e-06 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470143+0000", 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7241999999999999e-05 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.470156+0000", 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2462e-05 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496532+0000", 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026375778999999999 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496677+0000", 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145384 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.496829+0000", 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152337 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.330567677, 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:29:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:29:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.33052096599999, 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:29:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.675 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.33019085799999, 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:29:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:29:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.33016293599999, 2024-05-06T00:29:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:29:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.31929604699999, 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:29:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:29:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.661646245999997, 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.660461630999997, 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5979.0:3076 3.e 3:729546af:::rbd_object_map.110c3d37d7e3:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:02.121586+0000", 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.5013112629999998, 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026685349000000001, 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5979", 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/344188857", 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3076 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121586+0000", 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121586+0000", 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121586+0000", 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5700000000000001e-07 2024-05-06T00:29:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121592+0000", 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6960000000000002e-06 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121604+0000", 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1970000000000001e-05 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121612+0000", 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2169999999999994e-06 2024-05-06T00:29:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.140278+0000", 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018665360999999998 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.140386+0000", 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108476 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.140854+0000", 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046790200000000002 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.147917+0000", 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070633529999999996 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148062+0000", 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144124 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148192+0000", 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013016199999999999 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148271+0000", 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9231000000000001e-05 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5979.0:3077 3.e 3:729546af:::rbd_object_map.110c3d37d7e3:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-06T00:29:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:02.121626+0000", 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.5012703429999998, 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026902946, 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5979", 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/344188857", 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3077 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121626+0000", 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121626+0000", 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121627+0000", 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.06e-07 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121631+0000", 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9439999999999998e-06 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121640+0000", 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7229999999999993e-06 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121647+0000", 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5229999999999998e-06 2024-05-06T00:29:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.142015+0000", 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020367393000000001 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.142124+0000", 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010982899999999999 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.142592+0000", 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046780999999999998 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.147930+0000", 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053379339999999999 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148397+0000", 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046646099999999999 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148469+0000", 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1971999999999995e-05 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148529+0000", 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0751000000000002e-05 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5979.0:3078 3.e 3:729546af:::rbd_object_map.110c3d37d7e3:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:02.121661+0000", 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.501236247, 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027224011999999999, 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5979", 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/344188857", 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3078 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121661+0000", 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121661+0000", 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121661+0000", 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1999999999999999e-07 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121666+0000", 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6070000000000001e-06 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121675+0000", 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0329999999999997e-06 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121682+0000", 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6769999999999995e-06 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.143815+0000", 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022132912000000001 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.144016+0000", 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200979 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.144712+0000", 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00069567399999999997 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148550+0000", 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038382960000000002 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148635+0000", 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5082999999999998e-05 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148783+0000", 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014765199999999999 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148885+0000", 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101479 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5979.0:3079 3.e 3:729546af:::rbd_object_map.110c3d37d7e3:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-06T00:29:04.718 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:02.121831+0000", 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.5010657250000001, 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027368052, 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5979", 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/344188857", 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3079 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:04.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121831+0000", 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121831+0000", 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121832+0000", 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.376e-06 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121841+0000", 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0569999999999993e-06 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121859+0000", 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8967e-05 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121870+0000", 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0573e-05 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.145546+0000", 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023675952 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.145747+0000", 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000201189 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.146554+0000", 2024-05-06T00:29:04.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080715799999999999 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.149028+0000", 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024734380000000001 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.149051+0000", 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3146000000000002e-05 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.149135+0000", 2024-05-06T00:29:04.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3601000000000001e-05 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.149199+0000", 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4595000000000001e-05 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:04.724 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:04.725 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:04.641+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:29:04.726 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:04.725+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:04.727 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:04.827 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:04.826+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:04.845 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:04.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:04.845 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:04.846 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:04.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:04.906 INFO:teuthology.orchestra.run.smithi186.stdout:leaving image image_client.0-clone34 intact 2024-05-06T00:29:04.928 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:04.929+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:04.983 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:04.983 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:04.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:05.040 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:05.041+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:05.131 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:05.130+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:05.171 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 126.364999883, 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:29:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:29:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:29:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:29:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:29:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:29:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:29:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:29:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 126.36498367999999, 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356753462999997, 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:29:05.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356722093000002, 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:05.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:29:05.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356688304000002, 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:29:05.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:29:05.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:29:05.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356507848, 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:29:05.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:29:05.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356474204999998, 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356450422999998, 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:29:05.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:29:05.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356424146000002, 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:29:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:05.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356380905000002, 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:29:05.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:29:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:29:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:29:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:29:05.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:29:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:29:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356351275999998, 2024-05-06T00:29:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:29:05.228 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:29:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.229 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:29:05.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356318426000001, 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:29:05.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356286787999998, 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.239 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356253985999999, 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356221953000002, 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356189874000002, 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:05.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:29:05.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:29:05.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.356158565999998, 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:05.263 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.5160128489999991, 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.5146066999999999, 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.275 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.5145537420000004, 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.281 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:05.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:05.283 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:05.281+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:05.384 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:05.382+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:05.454 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:05.455 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:05.468 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:05.516 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:05.517+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:05.586 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:05.586+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:05.593 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:05.593 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:05.594 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:05.594 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:05.607 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:05.678 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-05-06T00:29:05.692 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:05.693+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:05.761 INFO:teuthology.orchestra.run.smithi186.stdout:5000 write 0xa9dfe00 thru 0xa9ee7cb (0xe9cc bytes) 2024-05-06T00:29:05.765 INFO:teuthology.orchestra.run.smithi186.stdout:5001 write 0x11fe800 thru 0x12021ce (0x39cf bytes) 2024-05-06T00:29:05.766 INFO:teuthology.orchestra.run.smithi186.stdout:5003 read 0x77c6200 thru 0x77c68f9 (0x6fa bytes) 2024-05-06T00:29:05.779 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747712804999999, 2024-05-06T00:29:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:29:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:29:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747692016000002, 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:29:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:29:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747671953999998, 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:29:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:29:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747652678000001, 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:29:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.74763256, 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747601150999998, 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:29:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:29:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747568239000003, 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747547652000002, 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:29:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747528271999997, 2024-05-06T00:29:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747483590999998, 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:05.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:29:05.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747418500000002, 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:29:05.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:05.832 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747386925999997, 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:05.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:05.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:29:05.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:29:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:29:05.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.838 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747364664999999, 2024-05-06T00:29:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:29:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:29:05.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.747343549, 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.844 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:05.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.089626922999997, 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:29:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.852 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.089216681000003, 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.150572087, 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.150500803, 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:29:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.150469169999999, 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:29:05.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.146250624, 2024-05-06T00:29:05.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:05.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:29:05.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:29:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:05.878 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:05.879 INFO:teuthology.orchestra.run.smithi186.stdout:5004 trunc from 0xc634a00 to 0x1882000 2024-05-06T00:29:05.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:05.881 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:05.877+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:05.929 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:29:05.929 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:29:05.930 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-06T00:29:05.930 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.580717 2024-05-06T00:29:05.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0.580716730852976 2024-05-06T00:29:05.971 INFO:teuthology.orchestra.run.smithi186.stdout:5006 write 0x50afa00 thru 0x50bf3b4 (0xf9b5 bytes) 2024-05-06T00:29:05.983 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:05.982+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:06.007 INFO:teuthology.orchestra.run.smithi186.stdout:5007 write 0x7715a00 thru 0x7725341 (0xf942 bytes) 2024-05-06T00:29:06.009 INFO:teuthology.orchestra.run.smithi186.stdout:5010 write 0x3848c00 thru 0x38495c6 (0x9c7 bytes) 2024-05-06T00:29:06.050 INFO:teuthology.orchestra.run.smithi186.stdout:5011 trunc from 0x7725400 to 0xe600600 2024-05-06T00:29:06.056 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:06.056 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:06.071 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:06.083 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:06.082+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:06.087 INFO:teuthology.orchestra.run.smithi186.stdout:5015 read 0x3ac5e00 thru 0x3ace446 (0x8647 bytes) 2024-05-06T00:29:06.103 INFO:teuthology.orchestra.run.smithi186.stdout:5018 trunc from 0xe600600 to 0x9917400 2024-05-06T00:29:06.130 INFO:teuthology.orchestra.run.smithi186.stdout:5019 read 0x2a19600 thru 0x2a1b244 (0x1c45 bytes) 2024-05-06T00:29:06.166 INFO:teuthology.orchestra.run.smithi186.stdout:5020 write 0xbbbbe00 thru 0xbbc0ab1 (0x4cb2 bytes) 2024-05-06T00:29:06.179 INFO:teuthology.orchestra.run.smithi186.stdout:5021 trunc from 0xbbc0c00 to 0x7bb3800 2024-05-06T00:29:06.183 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:06.182+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:06.191 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:06.192 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:06.206 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:06.220 INFO:teuthology.orchestra.run.smithi186.stdout:5027 read 0x5fc7c00 thru 0x5fcf8c4 (0x7cc5 bytes) 2024-05-06T00:29:06.222 INFO:teuthology.orchestra.run.smithi186.stdout:5029 write 0x1640e00 thru 0x16441bc (0x33bd bytes) 2024-05-06T00:29:06.223 INFO:teuthology.orchestra.run.smithi186.stdout:5030 write 0x79c9200 thru 0x79d33cd (0xa1ce bytes) 2024-05-06T00:29:06.233 INFO:teuthology.orchestra.run.smithi186.stdout:5033 trunc from 0x7bb3800 to 0xb3c9400 2024-05-06T00:29:06.261 INFO:teuthology.orchestra.run.smithi186.stdout:5034 flatten 2024-05-06T00:29:06.284 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:06.282+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:06.333 INFO:teuthology.orchestra.run.smithi186.stdout:5036 write 0x184cc00 thru 0x184e1ce (0x15cf bytes) 2024-05-06T00:29:06.335 INFO:teuthology.orchestra.run.smithi186.stdout:5037 read 0x8a3ca00 thru 0x8a40383 (0x3984 bytes) 2024-05-06T00:29:06.335 INFO:teuthology.orchestra.run.smithi186.stdout:5041 read 0x2f2ec00 thru 0x2f3b183 (0xc584 bytes) 2024-05-06T00:29:06.336 INFO:teuthology.orchestra.run.smithi186.stdout:5044 read 0x29adc00 thru 0x29bc8ee (0xecef bytes) 2024-05-06T00:29:06.336 INFO:teuthology.orchestra.run.smithi186.stdout:5045 read 0x93e3600 thru 0x93f158d (0xdf8e bytes) 2024-05-06T00:29:06.339 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 98.333805029000004, 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.340 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:06.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 98.333576457000007, 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:29:06.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 98.333499173000007, 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.349 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:29:06.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:29:06.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 98.333303192000002, 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:29:06.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.426193815999994, 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.361 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.426132937000006, 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.308576487000003, 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:29:06.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.370 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.308531758999997, 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:06.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:29:06.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.308493358, 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.375 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:29:06.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.308456096, 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:29:06.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.308419186999998, 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.308381269000002, 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.388 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:29:06.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:29:06.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:29:06.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.392 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.394 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:06.394 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.308344153999997, 2024-05-06T00:29:06.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:29:06.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.302563704999997, 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:29:06.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.302525621000001, 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:29:06.403 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:06.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.302487947000003, 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:29:06.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:29:06.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.293052437, 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:29:06.412 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:29:06.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.293020009999999, 2024-05-06T00:29:06.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:06.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.420 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.292996127999999, 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:06.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 62.292976197999998, 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:06.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:06.428 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:5046 trunc from 0xb3c9400 to 0x821c00 2024-05-06T00:29:06.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:06.431 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:06.429+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:06.447 INFO:teuthology.orchestra.run.smithi186.stdout:5049 trunc from 0x821c00 to 0x4f30400 2024-05-06T00:29:06.461 INFO:teuthology.orchestra.run.smithi186.stdout:5050 write 0x10ffa00 thru 0x1108bce (0x91cf bytes) 2024-05-06T00:29:06.483 INFO:teuthology.orchestra.run.smithi186.stdout:5051 trunc from 0x4f30400 to 0xd10b000 2024-05-06T00:29:06.505 INFO:teuthology.orchestra.run.smithi186.stdout:5054 read 0xb679c00 thru 0xb67e4c0 (0x48c1 bytes) 2024-05-06T00:29:06.507 INFO:teuthology.orchestra.run.smithi186.stdout:5055 write 0x2109200 thru 0x210beeb (0x2cec bytes) 2024-05-06T00:29:06.507 INFO:teuthology.orchestra.run.smithi186.stdout:5061 punch from 0x283bc00 to 0x283be00, (0x200 bytes) 2024-05-06T00:29:06.508 INFO:teuthology.orchestra.run.smithi186.stdout:5062 write 0x77c6a00 thru 0x77d1ed6 (0xb4d7 bytes) 2024-05-06T00:29:06.508 INFO:teuthology.orchestra.run.smithi186.stdout:5064 write 0x479c400 thru 0x479db4d (0x174e bytes) 2024-05-06T00:29:06.508 INFO:teuthology.orchestra.run.smithi186.stdout:5065 read 0xcd4e200 thru 0xcd58b9c (0xa99d bytes) 2024-05-06T00:29:06.508 INFO:teuthology.orchestra.run.smithi186.stdout:5067 read 0xbcf4200 thru 0xbd034e7 (0xf2e8 bytes) 2024-05-06T00:29:06.509 INFO:teuthology.orchestra.run.smithi186.stdout:5068 read 0x8e65800 thru 0x8e6dae3 (0x82e4 bytes) 2024-05-06T00:29:06.509 INFO:teuthology.orchestra.run.smithi186.stdout:5070 read 0x8e08600 thru 0x8e15249 (0xcc4a bytes) 2024-05-06T00:29:06.509 INFO:teuthology.orchestra.run.smithi186.stdout:5072 write 0x2572400 thru 0x257fc4e (0xd84f bytes) 2024-05-06T00:29:06.509 INFO:teuthology.orchestra.run.smithi186.stdout:5073 write 0x74b400 thru 0x75a1ab (0xedac bytes) 2024-05-06T00:29:06.531 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:06.529+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:06.541 INFO:teuthology.orchestra.run.smithi186.stdout:5074 write 0xea19000 thru 0xea282c4 (0xf2c5 bytes) 2024-05-06T00:29:06.558 INFO:teuthology.orchestra.run.smithi186.stdout:5076 trunc from 0xea28400 to 0x5735600 2024-05-06T00:29:06.576 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.5639.0:1315 3.14 e131/106 3:2a5a8b38:::rbd_object_map.110c5f062f8d:head v 131'377, mlcod=131'375)", 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:06.577517+0000", 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.00060434300000000002, 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062989800000000005, 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.577517+0000", 2024-05-06T00:29:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.577517+0000", 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.577518+0000", 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0699999999999999e-06 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.577526+0000", 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5550000000000003e-06 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.577545+0000", 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8671000000000001e-05 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.577555+0000", 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9140000000000003e-06 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.577639+0000", 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4664000000000001e-05 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.577674+0000", 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4891999999999997e-05 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-05-06T00:29:06.581 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:06.590 INFO:teuthology.orchestra.run.smithi186.stdout:5077 punch from 0x36b5200 to 0x36b6c00, (0x1a00 bytes) 2024-05-06T00:29:06.591 INFO:teuthology.orchestra.run.smithi186.stdout:5078 write 0x314da00 thru 0x31531b4 (0x57b5 bytes) 2024-05-06T00:29:06.592 INFO:teuthology.orchestra.run.smithi186.stdout:5080 read 0x3bc3c00 thru 0x3bca8d8 (0x6cd9 bytes) 2024-05-06T00:29:06.592 INFO:teuthology.orchestra.run.smithi186.stdout:5081 write 0x7faa00 thru 0x80a295 (0xf896 bytes) 2024-05-06T00:29:06.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:06.593 INFO:teuthology.orchestra.run.smithi186.stdout:5084 read 0x22c7400 thru 0x22d400e (0xcc0f bytes) 2024-05-06T00:29:06.599 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:06.603 INFO:teuthology.orchestra.run.smithi186.stdout:5085 trunc from 0x5735600 to 0xb554a00 2024-05-06T00:29:06.621 INFO:teuthology.orchestra.run.smithi186.stdout:5086 punch from 0x1bf5000 to 0x1c01a00, (0xca00 bytes) 2024-05-06T00:29:06.622 INFO:teuthology.orchestra.run.smithi186.stdout:5089 read 0x658d600 thru 0x6591fe5 (0x49e6 bytes) 2024-05-06T00:29:06.622 INFO:teuthology.orchestra.run.smithi186.stdout:5090 write 0x3f09e00 thru 0x3f0fd7f (0x5f80 bytes) 2024-05-06T00:29:06.625 INFO:teuthology.orchestra.run.smithi186.stdout:5091 trunc from 0xb554a00 to 0xd1d2800 2024-05-06T00:29:06.641 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:06.642+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:06.642 INFO:teuthology.orchestra.run.smithi186.stdout:5094 write 0x8c48600 thru 0x8c4d987 (0x5388 bytes) 2024-05-06T00:29:06.643 INFO:teuthology.orchestra.run.smithi186.stdout:5095 write 0x2586600 thru 0x258d5a6 (0x6fa7 bytes) 2024-05-06T00:29:06.643 INFO:teuthology.orchestra.run.smithi186.stdout:5096 punch from 0xdec800 to 0xdf4e00, (0x8600 bytes) 2024-05-06T00:29:06.654 INFO:teuthology.orchestra.run.smithi186.stdout:5101 trunc from 0xd1d2800 to 0x76d3600 2024-05-06T00:29:06.697 INFO:teuthology.orchestra.run.smithi186.stdout:5102 write 0xda76200 thru 0xda7fdd0 (0x9bd1 bytes) 2024-05-06T00:29:06.697 INFO:teuthology.orchestra.run.smithi186.stdout:5104 write 0xa2ca200 thru 0xa2d9463 (0xf264 bytes) 2024-05-06T00:29:06.697 INFO:teuthology.orchestra.run.smithi186.stdout:5105 punch from 0x46aa00 to 0x46fa00, (0x5000 bytes) 2024-05-06T00:29:06.698 INFO:teuthology.orchestra.run.smithi186.stdout:5106 clone 36 order 25 su 131072 sc 7 2024-05-06T00:29:06.720 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.5 primary-affinity to 0.580717 (8380572) 2024-05-06T00:29:06.732 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:06.730+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:06.734 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:06.734 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:06.734 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:06.734 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:06.734 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:06.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:29:06.748 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:06.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:06.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:06.794 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:06.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:06.810 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:06.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:06.829 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:06.836 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:06.838+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:06.846 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:06.846 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:06.865 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:06.866 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:06.910 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:06.910 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:06.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:06.934 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:06.933+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:07.035 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:07.033+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:07.082 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:07.082 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:07.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:07.141 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:07.142+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:07.216 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:07.217 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:07.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:07.273 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:07.274+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:07.338 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:07.338+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:07.395 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:07.395 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:07.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:07.453 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:07.454+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:07.539 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:07.538+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:07.577 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:07.577 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:07.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:07.677 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:07.678+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:07.728 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:29:07.741 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:07.741+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:07.768 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:07.768 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:07.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:07.842 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:07.841+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:07.943 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:07.943 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:07.944 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:07.942+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:07.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:08.045 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:08.045+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:08.089 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:08.090 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:08.104 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:08.145 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:08.146+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:08.246 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:08.246+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:08.274 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:08.275 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:08.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:08.348 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:08.345+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:08.418 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:08.418 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:08.433 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:08.450 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:08.450+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:08.521 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #34, image image_client.0-clone34 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent35 2024-05-06T00:29:08.550 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:08.549+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:08.596 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:08.596 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:08.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:08.652 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:08.653+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:08.752 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:08.749+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:08.792 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:08.792 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:08.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:08.885 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:08.892 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:08.893+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:08.955 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:08.953+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:09.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:09.040 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:09.041 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:09.058 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:09.059 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:09.059 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:09.058+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:09.076 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:09.076 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:09.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:09.100 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:09.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:09.122 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:09.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:09.143 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:09.160 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:09.158+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:09.172 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:09.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:09.172 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:09.172 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:09.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:09.263 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:09.262+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:09.358 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:09.359 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:09.359 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:09.364 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:09.362+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:09.375 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:09.466 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:09.466+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:09.504 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.497600127, 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:29:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.497371555, 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:09.513 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.497294271, 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:09.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:29:09.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:29:09.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.49709829, 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.518 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:29:09.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:29:09.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:29:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.589988914000003, 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:09.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:29:09.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:29:09.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:29:09.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.589928035, 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.472371585000005, 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:09.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:29:09.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:29:09.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:29:09.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.472326856999999, 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:29:09.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:29:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.472288456000001, 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:29:09.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.472251193999995, 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.472214285000007, 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.472176367000003, 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:29:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.472139252000005, 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.466358803000006, 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.466320718999995, 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:09.574 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.466283044999997, 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:29:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.456847534999994, 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:29:09.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:09.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:29:09.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.456815108000001, 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.583 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:29:09.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.456791226000007, 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.590 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.456771295999999, 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:09.594 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:09.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:09.596 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:09.594+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:09.697 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:09.697+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:09.728 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:09.729 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:09.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:09.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:29:09.770 INFO:teuthology.orchestra.run.smithi186.stdout:5107 read 0xda05800 thru 0xda0bc91 (0x6492 bytes) 2024-05-06T00:29:09.770 INFO:teuthology.orchestra.run.smithi186.stdout:5110 read 0xd0f3c00 thru 0xd102c4f (0xf050 bytes) 2024-05-06T00:29:09.771 INFO:teuthology.orchestra.run.smithi186.stdout:5111 clone 37 order 25 su 524288 sc 14 2024-05-06T00:29:09.800 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:09.801+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:09.866 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:09.866 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:09.866 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:09.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:09.925 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:09.926+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:10.000 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:09.998+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:10.006 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:10.006 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:10.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:10.006 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:10.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.006 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:29:10.006 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:10.006 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 160.401028, 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:29:10.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.013 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.128379301, 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:29:10.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:29:10.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.128344125, 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:29:10.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.128255446, 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.026 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:29:10.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:29:10.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.12815350299999, 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:29:10.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.128044401, 2024-05-06T00:29:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:29:10.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:29:10.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:29:10.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:29:10.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.127946542, 2024-05-06T00:29:10.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:29:10.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:29:10.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:29:10.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:29:10.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:29:10.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:29:10.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:29:10.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:29:10.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:29:10.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:29:10.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:29:10.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:29:10.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.054 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:10.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.127881492, 2024-05-06T00:29:10.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:29:10.055 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:29:10.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:29:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.127788846, 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:29:10.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:29:10.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.127716842, 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:10.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:29:10.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:29:10.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.36825535600001, 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.075 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.075 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:29:10.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.077 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:29:10.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:29:10.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:29:10.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:29:10.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:29:10.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.368236108, 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.083 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:29:10.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:29:10.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.36818598799999, 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.090 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:10.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:29:10.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.368164937, 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:29:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:29:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.368149618, 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:29:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:29:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:29:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:29:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:29:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:29:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:29:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:29:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:29:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:29:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:29:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:29:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:29:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:29:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:29:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:29:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:29:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:29:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:29:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:29:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:29:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:29:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:29:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.36813273600001, 2024-05-06T00:29:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:29:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:29:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:29:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:29:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:29:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:29:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:29:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:29:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:29:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:29:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:29:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:29:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:29:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:29:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:29:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:29:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:29:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:29:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:29:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:29:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:29:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:29:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:29:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:29:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.3681144, 2024-05-06T00:29:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:29:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:29:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:29:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:29:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.368098398, 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:29:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:29:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.36801203500001, 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:29:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:29:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:29:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.35804955899999, 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:29:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.153 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:10.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-06T00:29:10.097+0000 7effa6ffd640 0 --2- 172.21.15.78:0/3748571220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effa80a0cd0 0x7effa8133840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-05-06T00:29:10.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:10.156 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:10.153+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:10.255 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:10.253+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:10.324 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:10.324 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:10.324 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:10.324 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:10.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:10.356 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:10.357+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:10.457 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:10.457 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:10.457 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:10.457+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:10.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:10.558 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:10.558+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:10.596 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:10.596 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.934122182999999, 2024-05-06T00:29:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954880677999995, 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954806458999997, 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954744559999995, 2024-05-06T00:29:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954684960999998, 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954600826000004, 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954535312999994, 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954477451000002, 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954380856, 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954303131000003, 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954230344999999, 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954152854, 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.954058724999996, 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.953975763000003, 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.953901658000007, 2024-05-06T00:29:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.953802468999996, 2024-05-06T00:29:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.953739317, 2024-05-06T00:29:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.953639052, 2024-05-06T00:29:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.953549679999995, 2024-05-06T00:29:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.707690997, 2024-05-06T00:29:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:10.738 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:10.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:10.740 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:10.738+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:10.749 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:29:10.841 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:10.841+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:10.888 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone36 from 0xda7fe00 (overlap 0xda7fe00) to 0x0 2024-05-06T00:29:10.920 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:10.920 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:10.920 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:10.920 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:10.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:10.985 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:10.986+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:11.043 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:11.042+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:11.060 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:11.060 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:11.061 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:11.061 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:11.061 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:11.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:11.144 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:11.141+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:11.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:11.245 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:11.245+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:11.248 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2794 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 249856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.787591431, 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026871237999999999, 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2794 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459176+0000", 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459178+0000", 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459183+0000", 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0429999999999998e-06 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459198+0000", 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5786000000000001e-05 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.459239+0000", 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0398000000000002e-05 2024-05-06T00:29:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485611+0000", 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026372715000000001 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.485790+0000", 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178832 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486048+0000", 2024-05-06T00:29:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025740099999999998 2024-05-06T00:29:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2863 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 151552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.786630395, 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026891953999999999, 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2863 2024-05-06T00:29:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460137+0000", 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460139+0000", 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-06 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460144+0000", 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.293e-06 2024-05-06T00:29:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460165+0000", 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1353e-05 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460179+0000", 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3869000000000001e-05 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486901+0000", 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02672213 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.486979+0000", 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7354000000000005e-05 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487029+0000", 2024-05-06T00:29:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0695000000000002e-05 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2864 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 184320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.786570544, 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027314798000000001, 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2864 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460197+0000", 2024-05-06T00:29:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460198+0000", 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.08e-06 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460204+0000", 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460221+0000", 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7218999999999999e-05 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460255+0000", 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3618999999999998e-05 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487112+0000", 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026857161000000001 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487319+0000", 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000206877 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487512+0000", 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019329199999999999 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2865 3.16 3:6bafe9a8:::rbd_data.110cc6eb3028.0000000000000009:c [read 217088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.78649486099999, 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027389264999999999, 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2865 2024-05-06T00:29:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460273+0000", 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.460274+0000", 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.257e-06 2024-05-06T00:29:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465840+0000", 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055656910000000002 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465867+0000", 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6919000000000001e-05 2024-05-06T00:29:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.465885+0000", 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8320000000000001e-05 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487533+0000", 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021647845999999998 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487590+0000", 2024-05-06T00:29:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7058999999999997e-05 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.487662+0000", 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2173000000000002e-05 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:2970 3.e 3:7057030d:::rbd_data.110cc6eb3028.0000000000000000:c [read 258048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.778682932, 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027146446000000001, 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2970 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468085+0000", 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468086+0000", 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.063e-06 2024-05-06T00:29:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468091+0000", 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3879999999999999e-06 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468109+0000", 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7178999999999999e-05 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.468121+0000", 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.292e-05 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.494912+0000", 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026790074000000001 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495074+0000", 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161986 2024-05-06T00:29:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.495231+0000", 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015783600000000001 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2947 3.e 3:725816fb:::rbd_data.110cd5ddb45c.00000000000000a1:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.95443874599999, 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03119622, 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2947 2024-05-06T00:29:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292329+0000", 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292330+0000", 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2e-07 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292333+0000", 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2739999999999999e-06 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292340+0000", 2024-05-06T00:29:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9040000000000003e-06 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292345+0000", 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0889999999999999e-06 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292493+0000", 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148159 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292563+0000", 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9548000000000006e-05 2024-05-06T00:29:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293195+0000", 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063233899999999999 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.298113+0000", 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00491764 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323431+0000", 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025318202000000001 2024-05-06T00:29:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323480+0000", 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9109e-05 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323525+0000", 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5435999999999997e-05 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2949 3.e 3:72796e3e:::rbd_data.110cd5ddb45c.00000000000000a4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.954392035, 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031267436000000003, 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2949 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292376+0000", 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999998e-07 2024-05-06T00:29:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292379+0000", 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6539999999999999e-06 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292385+0000", 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0190000000000003e-06 2024-05-06T00:29:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292390+0000", 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7829999999999998e-06 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296583+0000", 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041933939999999996 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296656+0000", 2024-05-06T00:29:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2849000000000007e-05 2024-05-06T00:29:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.296840+0000", 2024-05-06T00:29:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183499 2024-05-06T00:29:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.300538+0000", 2024-05-06T00:29:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036982579999999998 2024-05-06T00:29:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323561+0000", 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023023422000000002 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323611+0000", 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9796999999999998e-05 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323643+0000", 2024-05-06T00:29:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2243999999999998e-05 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2956 3.e 3:71f26ebd:::rbd_data.110cd5ddb45c.00000000000000a8:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.954061927, 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031062781000000001, 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2956 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292706+0000", 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.13e-07 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292710+0000", 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1860000000000001e-06 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292716+0000", 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.793e-06 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292722+0000", 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.648e-06 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297656+0000", 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049340419999999996 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297730+0000", 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4326999999999995e-05 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.297904+0000", 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017315000000000001 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.301308+0000", 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034046229999999999 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323687+0000", 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022378695000000001 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323735+0000", 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8510000000000001e-05 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323769+0000", 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3293999999999997e-05 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:2957 3.1c 3:3f894501:::rbd_data.110cd5ddb45c.00000000000000d0:head [stat,delete] snapc e=[e] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.954034005, 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030488764000000002, 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2957 2024-05-06T00:29:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292734+0000", 2024-05-06T00:29:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3300000000000002e-07 2024-05-06T00:29:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292737+0000", 2024-05-06T00:29:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.481e-06 2024-05-06T00:29:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292744+0000", 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1049999999999997e-06 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.292749+0000", 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2299999999999999e-06 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293280+0000", 2024-05-06T00:29:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000530552 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.293471+0000", 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019164599999999999 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.323223+0000", 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029751217 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4932.0:3042 3.13 3.895d0413 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.943167116, 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027366475000000001, 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4932", 2024-05-06T00:29:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3604624518", 2024-05-06T00:29:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3042 2024-05-06T00:29:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303601+0000", 2024-05-06T00:29:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:29:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303604+0000", 2024-05-06T00:29:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8399999999999999e-06 2024-05-06T00:29:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303612+0000", 2024-05-06T00:29:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7389999999999999e-06 2024-05-06T00:29:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.303618+0000", 2024-05-06T00:29:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4740000000000002e-06 2024-05-06T00:29:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.327093+0000", 2024-05-06T00:29:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023474611999999999 2024-05-06T00:29:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:24.330967+0000", 2024-05-06T00:29:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038741859999999999 2024-05-06T00:29:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.285517315, 2024-05-06T00:29:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.2843327, 2024-05-06T00:29:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:11.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:11.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:11.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5979.0:3077 3.e 3:729546af:::rbd_object_map.110c3d37d7e3:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-06T00:29:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:02.121626+0000", 2024-05-06T00:29:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.1251414119999996, 2024-05-06T00:29:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026902946, 2024-05-06T00:29:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5979", 2024-05-06T00:29:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/344188857", 2024-05-06T00:29:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3077 2024-05-06T00:29:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121626+0000", 2024-05-06T00:29:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121626+0000", 2024-05-06T00:29:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121627+0000", 2024-05-06T00:29:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.06e-07 2024-05-06T00:29:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121631+0000", 2024-05-06T00:29:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9439999999999998e-06 2024-05-06T00:29:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121640+0000", 2024-05-06T00:29:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7229999999999993e-06 2024-05-06T00:29:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121647+0000", 2024-05-06T00:29:11.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5229999999999998e-06 2024-05-06T00:29:11.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.142015+0000", 2024-05-06T00:29:11.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020367393000000001 2024-05-06T00:29:11.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.142124+0000", 2024-05-06T00:29:11.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010982899999999999 2024-05-06T00:29:11.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:11.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.142592+0000", 2024-05-06T00:29:11.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046780999999999998 2024-05-06T00:29:11.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:11.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.147930+0000", 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053379339999999999 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148397+0000", 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046646099999999999 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148469+0000", 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1971999999999995e-05 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148529+0000", 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0751000000000002e-05 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5979.0:3078 3.e 3:729546af:::rbd_object_map.110c3d37d7e3:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:02.121661+0000", 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.1251073159999994, 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027224011999999999, 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.343 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5979", 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/344188857", 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3078 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121661+0000", 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121661+0000", 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121661+0000", 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1999999999999999e-07 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121666+0000", 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6070000000000001e-06 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121675+0000", 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0329999999999997e-06 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121682+0000", 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6769999999999995e-06 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.143815+0000", 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022132912000000001 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.144016+0000", 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200979 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.144712+0000", 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00069567399999999997 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148550+0000", 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038382960000000002 2024-05-06T00:29:11.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148635+0000", 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5082999999999998e-05 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148783+0000", 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014765199999999999 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.148885+0000", 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101479 2024-05-06T00:29:11.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5979.0:3079 3.e 3:729546af:::rbd_object_map.110c3d37d7e3:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:02.121831+0000", 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.1249367939999999, 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027368052, 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5979", 2024-05-06T00:29:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/344188857", 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3079 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121831+0000", 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121831+0000", 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121832+0000", 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.376e-06 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121841+0000", 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0569999999999993e-06 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121859+0000", 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8967e-05 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.121870+0000", 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0573e-05 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.145546+0000", 2024-05-06T00:29:11.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023675952 2024-05-06T00:29:11.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.145747+0000", 2024-05-06T00:29:11.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000201189 2024-05-06T00:29:11.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:11.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.146554+0000", 2024-05-06T00:29:11.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080715799999999999 2024-05-06T00:29:11.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:11.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.149028+0000", 2024-05-06T00:29:11.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024734380000000001 2024-05-06T00:29:11.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:11.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.149051+0000", 2024-05-06T00:29:11.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3146000000000002e-05 2024-05-06T00:29:11.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:11.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.149135+0000", 2024-05-06T00:29:11.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3601000000000001e-05 2024-05-06T00:29:11.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:02.149199+0000", 2024-05-06T00:29:11.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4595000000000001e-05 2024-05-06T00:29:11.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.357 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.357 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5639.0:932 3.1c 3:3840fe09:::rbd_header.110c5f062f8d:head [watch reconnect cookie 139727164508032 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-05-06T00:29:11.358 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:06.283637+0000", 2024-05-06T00:29:11.358 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9631312249999997, 2024-05-06T00:29:11.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028419686999999999, 2024-05-06T00:29:11.358 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.358 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.358 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5639", 2024-05-06T00:29:11.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3387867120", 2024-05-06T00:29:11.359 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 932 2024-05-06T00:29:11.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.359 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283637+0000", 2024-05-06T00:29:11.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283637+0000", 2024-05-06T00:29:11.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283637+0000", 2024-05-06T00:29:11.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2200000000000004e-07 2024-05-06T00:29:11.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283640+0000", 2024-05-06T00:29:11.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9249999999999999e-06 2024-05-06T00:29:11.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283648+0000", 2024-05-06T00:29:11.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5750000000000003e-06 2024-05-06T00:29:11.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283654+0000", 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3300000000000004e-06 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.308785+0000", 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025130611000000001 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.308879+0000", 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4096000000000002e-05 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.312056+0000", 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031775280000000002 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5639.0:936 3.1c 3:3840fe09:::rbd_header.110c5f062f8d:head [notify cookie 139726828908288] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:06.283773+0000", 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9629949920000005, 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027143183000000001, 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5639", 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3387867120", 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 936 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.366 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283773+0000", 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283773+0000", 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283774+0000", 2024-05-06T00:29:11.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0100000000000001e-06 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283778+0000", 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8330000000000004e-06 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283788+0000", 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0574000000000001e-05 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283795+0000", 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9179999999999997e-06 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.310628+0000", 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026832812000000001 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.310734+0000", 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010592 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.310916+0000", 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018211599999999999 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5639.0:937 3.1c 3:3840fe09:::rbd_header.110c5f062f8d:head [notify cookie 139726828919856] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:06.283803+0000", 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9629647869999998, 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027473345999999999, 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.370 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5639", 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3387867120", 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 937 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283803+0000", 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283803+0000", 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283804+0000", 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5499999999999997e-07 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283807+0000", 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.315e-06 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283814+0000", 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5980000000000002e-06 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283820+0000", 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.806e-06 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.310972+0000", 2024-05-06T00:29:11.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027152652999999999 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.311072+0000", 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100059 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.311276+0000", 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020416 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5639.0:938 3.1c 3:3840fe09:::rbd_header.110c5f062f8d:head [notify cookie 139726828931536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:06.283828+0000", 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9629402530000002, 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027887636, 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5639", 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3387867120", 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 938 2024-05-06T00:29:11.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283828+0000", 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283828+0000", 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283828+0000", 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8800000000000003e-07 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283831+0000", 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2550000000000002e-06 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283838+0000", 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5300000000000002e-06 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283844+0000", 2024-05-06T00:29:11.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0700000000000003e-06 2024-05-06T00:29:11.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.311410+0000", 2024-05-06T00:29:11.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027566493000000001 2024-05-06T00:29:11.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.311527+0000", 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116476 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.311715+0000", 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018832399999999999 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5639.0:939 3.1c 3:3840fe09:::rbd_header.110c5f062f8d:head [notify cookie 139726828943936] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:06.283855+0000", 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9629127430000004, 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028204942, 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:11.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5639", 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3387867120", 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 939 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283855+0000", 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283855+0000", 2024-05-06T00:29:11.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283856+0000", 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0800000000000004e-07 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283862+0000", 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6019999999999999e-06 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283873+0000", 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1127999999999999e-05 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.283879+0000", 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999999e-06 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.311784+0000", 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027905685 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.311869+0000", 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4504000000000003e-05 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:06.312060+0000", 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019110500000000001 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:11.384 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:11.386 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:11.387 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:11.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:11.388 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:11.389 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:11.386+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:11.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:11.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:11.390 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:11.391 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:11.392 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:11.397 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:11.418 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:11.419 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:11.474 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #35, image image_client.0-clone35 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent36 2024-05-06T00:29:11.487 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:11.486+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:11.510 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:11.510 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:11.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:11.588 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:11.589+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:11.647 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:11.647 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:11.662 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:11.704 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:11.705+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:11.750 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:29:11.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:29:11.750 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-06T00:29:11.750 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.833284 2024-05-06T00:29:11.750 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 0.8332842276803453 2024-05-06T00:29:11.790 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:11.790+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:11.826 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:11.826 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:11.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:11.892 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:11.893+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:11.972 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:11.972 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:11.988 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:11.991 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:11.990+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:12.091 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:12.090+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:12.157 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:12.157 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:12.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:12.216 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:12.217+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:12.293 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:12.294+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:12.295 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:12.295 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:12.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:12.404 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:12.405+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:12.485 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:12.485 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:12.496 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:12.494+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:12.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:12.596 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:12.598+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:12.661 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:12.661 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:12.675 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:12.720 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:12.721+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.4 primary-affinity to 0.833284 (8546102) 2024-05-06T00:29:12.779 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:29:12.799 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:12.798+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:12.802 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:12.802 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:12.814 INFO:teuthology.orchestra.run.smithi186.stdout:5112 read 0x461e200 thru 0x4627cd0 (0x9ad1 bytes) 2024-05-06T00:29:12.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:12.820 INFO:teuthology.orchestra.run.smithi186.stdout:5113 read 0x59c2a00 thru 0x59cc3ac (0x99ad bytes) 2024-05-06T00:29:12.900 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:12.901+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:12.951 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:12.951 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:12.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:13.009 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:13.010+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:13.084 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:13.084 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:13.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:13.102 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:13.101+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:13.203 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:13.201+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:13.260 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:13.260 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:13.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:13.321 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:13.322+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:13.393 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:13.393 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:13.405 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:13.406+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:13.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:13.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:13.506 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:13.505+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:13.579 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:13.579 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:13.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:13.595 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:13.614 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:13.614 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:13.629 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:13.630 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:13.648 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:13.650+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:13.649 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:13.649 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:13.667 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:13.668 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:13.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:13.688 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:13.707 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:13.706+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:13.715 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.715 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.055230800000004, 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:13.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075989294999999, 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.720 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:13.724 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075915076000001, 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075853176999999, 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:13.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:13.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:13.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075793578000003, 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:13.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:13.737 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075709442999994, 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:13.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:13.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075643929999998, 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:13.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075586068000007, 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:13.746 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075489473000005, 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:13.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:13.755 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075411747999993, 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:13.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075338962000004, 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:13.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075261471000005, 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:13.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075167342, 2024-05-06T00:29:13.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:13.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:13.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.772 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075084380000007, 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.776 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.075010274999997, 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:13.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.074911086, 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:13.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:13.785 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.074847934000005, 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:13.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.074747669000004, 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.074658296999999, 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:13.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:13.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:13.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:13.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.828799614000005, 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:13.798 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:13.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:13.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:13.802 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:13.802 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:13.803 INFO:teuthology.orchestra.run.smithi186.stdout:5115 trunc from 0xda7fe00 to 0xc6b6600 2024-05-06T00:29:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:29:13.804 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:13.856 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:13.854+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:13.912 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:13.914+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:13.922 INFO:teuthology.orchestra.run.smithi186.stdout:5117 read 0xa936e00 thru 0xa944fba (0xe1bb bytes) 2024-05-06T00:29:13.950 INFO:teuthology.orchestra.run.smithi186.stdout:5118 write 0xe8c2600 thru 0xe8cfb19 (0xd51a bytes) 2024-05-06T00:29:13.995 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:13.995 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:14.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:14.027 INFO:teuthology.orchestra.run.smithi186.stdout:5119 trunc from 0xe8cfc00 to 0x4bada00 2024-05-06T00:29:14.054 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:14.054+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:14.067 INFO:teuthology.orchestra.run.smithi186.stdout:5121 read 0x379bc00 thru 0x37a85a3 (0xc9a4 bytes) 2024-05-06T00:29:14.069 INFO:teuthology.orchestra.run.smithi186.stdout:5122 flatten 2024-05-06T00:29:14.105 INFO:teuthology.orchestra.run.smithi186.stdout:5124 trunc from 0x4bada00 to 0x3791400 2024-05-06T00:29:14.112 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:14.113+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:14.130 INFO:teuthology.orchestra.run.smithi186.stdout:5125 read 0x1ea6400 thru 0x1eabd9f (0x59a0 bytes) 2024-05-06T00:29:14.163 INFO:teuthology.orchestra.run.smithi186.stdout:5129 write 0x76a4000 thru 0x76acaa5 (0x8aa6 bytes) 2024-05-06T00:29:14.183 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:14.183 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:14.183 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:14.183 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:14.184 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:14.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:14.204 INFO:teuthology.orchestra.run.smithi186.stdout:5131 write 0xd89cc00 thru 0xd8a002f (0x3430 bytes) 2024-05-06T00:29:14.204 INFO:teuthology.orchestra.run.smithi186.stdout:5132 write 0xce80000 thru 0xce85f3f (0x5f40 bytes) 2024-05-06T00:29:14.206 INFO:teuthology.orchestra.run.smithi186.stdout:5133 clone 38 order 18 su 131072 sc 14 2024-05-06T00:29:14.213 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:14.213+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:14.314 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:14.313+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:14.328 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:14.328 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.365129932000002, 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:14.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.363945317000002, 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.335 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:14.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.48052427199999997, 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:29:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.48048705899999999, 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:29:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.480398253, 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:29:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.48033806099999998, 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:29:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:29:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.48028375800000001, 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.480217118, 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:29:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:29:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.48015484600000002, 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:29:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.48009584999999999, 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:29:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.47369112899999999, 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:29:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.47364239499999999, 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:29:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:29:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.47359814300000003, 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:29:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.473527111, 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:29:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:29:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.47347629600000002, 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:29:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.46473714799999999, 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.464714134, 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:29:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:29:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.464691884, 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.46458203399999998, 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.46451798399999999, 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:14.425 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:14.426 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:14.426+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:14.426 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:14.527 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:14.525+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:14.559 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:14.559 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:14.572 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:14.664 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:14.665+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:14.729 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:14.729+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:14.738 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:14.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:14.738 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:14.739 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:14.739 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:14.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:14.840 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:14.841+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:14.925 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 136.11951914900001, 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:29:14.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:14.931 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 136.11950294600001, 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:14.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.111272729000007, 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:14.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.111241359000005, 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:29:14.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:29:14.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:29:14.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.111207570000005, 2024-05-06T00:29:14.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:29:14.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:29:14.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:29:14.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.111027113999995, 2024-05-06T00:29:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:29:14.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:29:14.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110993471, 2024-05-06T00:29:14.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:29:14.958 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.958 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.958 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.958 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:14.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:29:14.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110969689000001, 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:29:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:29:14.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.969 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110943411999997, 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:29:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110900170999997, 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:29:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:29:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:29:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110870542000001, 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:29:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:29:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110837692000004, 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110806053999994, 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:14.996 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110773252000001, 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:14.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:14.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.001 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110741219000005, 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.006 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110709139999997, 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:15.011 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.110677831999993, 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:29:15.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:29:15.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.270532115000002, 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:15.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:29:15.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:29:15.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.269125966000001, 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:29:15.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:29:15.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.269073007999999, 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:15.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.033 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:15.035 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:15.035 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:15.034+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:15.135 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:15.134+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:15.209 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:15.209 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:15.209 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:15.209 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:15.223 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:15.237 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:15.238+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:15.338 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:15.337+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:15.346 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:15.346 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:15.347 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:15.347 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:15.347 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:15.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:15.438 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:15.438+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:15.507 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #36, image image_client.0-clone36 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent37 2024-05-06T00:29:15.533 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:15.533 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505328500999994, 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:29:15.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:15.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:29:15.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:29:15.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:29:15.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505307712000004, 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:15.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.543 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.50528765, 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:29:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:29:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505268373999996, 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:29:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505248256000002, 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:29:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505216847, 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:29:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:29:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.562 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505183935000005, 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:29:15.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505163347999996, 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:29:15.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.569 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:15.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:29:15.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:29:15.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505143967999999, 2024-05-06T00:29:15.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:15.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:29:15.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505099286999993, 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:15.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505034195999997, 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:29:15.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.505002622000006, 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:15.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:29:15.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:29:15.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.504980360999994, 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.599 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 71.504959244999995, 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:29:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:29:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:29:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.847242618999999, 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:29:15.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.609 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.846832376999998, 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:29:15.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:29:15.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:29:15.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:29:15.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:29:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.908187782999999, 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:29:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:29:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.908116498999998, 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:29:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:29:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:29:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.908084865999999, 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:29:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:29:15.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:15.631 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.903866319999999, 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:15.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:15.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:29:15.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:29:15.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:15.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:29:15.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:15.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:15.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:15.638 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:15.639 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:15.640 INFO:teuthology.orchestra.run.smithi186.stdout:5134 read 0x4bed800 thru 0x4bf955e (0xbd5f bytes) 2024-05-06T00:29:15.640 INFO:teuthology.orchestra.run.smithi186.stdout:5135 punch from 0x47c9800 to 0x47cf000, (0x5800 bytes) 2024-05-06T00:29:15.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:15.641 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:15.638+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:15.691 INFO:teuthology.orchestra.run.smithi186.stdout:5136 trunc from 0xd8a0200 to 0x2788a00 2024-05-06T00:29:15.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:15.752 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:15.753+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:15.753 INFO:teuthology.orchestra.run.smithi186.stdout:5137 read 0x1609000 thru 0x16106b6 (0x76b7 bytes) 2024-05-06T00:29:15.780 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:15.780 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:15.780 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:15.780 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:15.796 INFO:teuthology.orchestra.run.smithi186.stdout:5140 write 0xc9f4600 thru 0xca01a9c (0xd49d bytes) 2024-05-06T00:29:15.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:15.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:29:15.810 INFO:teuthology.orchestra.run.smithi186.stdout:5141 trunc from 0xca01c00 to 0x68a1000 2024-05-06T00:29:15.842 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:15.841+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:15.848 INFO:teuthology.orchestra.run.smithi186.stdout:5142 write 0x4925200 thru 0x4926d71 (0x1b72 bytes) 2024-05-06T00:29:15.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:15.862 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:15.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:15.899 INFO:teuthology.orchestra.run.smithi186.stdout:5144 write 0xe76fc00 thru 0xe77be0e (0xc20f bytes) 2024-05-06T00:29:15.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:15.905 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:15.925 INFO:teuthology.orchestra.run.smithi186.stdout:5146 trunc from 0xe77c000 to 0x3870800 2024-05-06T00:29:15.927 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:15.927 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:15.943 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:15.942+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:15.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:15.952 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:15.968 INFO:teuthology.orchestra.run.smithi186.stdout:5147 read 0x1bcb000 thru 0x1bcdb50 (0x2b51 bytes) 2024-05-06T00:29:15.970 INFO:teuthology.orchestra.run.smithi186.stdout:5149 write 0x271a000 thru 0x272565d (0xb65e bytes) 2024-05-06T00:29:15.974 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:15.974 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:15.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:15.978 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:15.981 INFO:teuthology.orchestra.run.smithi186.stdout:5152 trunc from 0x3870800 to 0x74dc600 2024-05-06T00:29:15.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:16.015 INFO:teuthology.orchestra.run.smithi186.stdout:5154 write 0x7b44800 thru 0x7b50d01 (0xc502 bytes) 2024-05-06T00:29:16.017 INFO:teuthology.orchestra.run.smithi186.stdout:5156 punch from 0x5df3c00 to 0x5dfa400, (0x6800 bytes) 2024-05-06T00:29:16.018 INFO:teuthology.orchestra.run.smithi186.stdout:5158 read 0x5e74400 thru 0x5e782bd (0x3ebe bytes) 2024-05-06T00:29:16.021 INFO:teuthology.orchestra.run.smithi186.stdout:5159 trunc from 0x7b50e00 to 0x986cc00 2024-05-06T00:29:16.038 INFO:teuthology.orchestra.run.smithi186.stdout:5161 read 0x4d7dc00 thru 0x4d85da8 (0x81a9 bytes) 2024-05-06T00:29:16.040 INFO:teuthology.orchestra.run.smithi186.stdout:5162 write 0x38c6800 thru 0x38d57f6 (0xeff7 bytes) 2024-05-06T00:29:16.040 INFO:teuthology.orchestra.run.smithi186.stdout:5165 read 0x8a1e600 thru 0x8a2b889 (0xd28a bytes) 2024-05-06T00:29:16.041 INFO:teuthology.orchestra.run.smithi186.stdout:5166 punch from 0x6a10e00 to 0x6a15800, (0x4a00 bytes) 2024-05-06T00:29:16.049 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:16.049+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:16.063 INFO:teuthology.orchestra.run.smithi186.stdout:5168 write 0xaa96200 thru 0xaa9b60f (0x5410 bytes) 2024-05-06T00:29:16.064 INFO:teuthology.orchestra.run.smithi186.stdout:5170 write 0x32ab200 thru 0x32b4a9a (0x989b bytes) 2024-05-06T00:29:16.064 INFO:teuthology.orchestra.run.smithi186.stdout:5171 read 0x322b200 thru 0x32316c9 (0x64ca bytes) 2024-05-06T00:29:16.064 INFO:teuthology.orchestra.run.smithi186.stdout:5173 clone 39 order 18 su 32768 sc 10 2024-05-06T00:29:16.145 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:16.146+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:16.168 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:16.168 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:16.182 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:16.246 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:16.245+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:16.316 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:16.316 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:16.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:16.377 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:16.378+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:16.448 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:16.450+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:16.454 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:16.454 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:16.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:16.550 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:16.549+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:16.642 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:16.642 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:16.651 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:16.649+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:16.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:16.752 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:16.749+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:16.774 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:16.774 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:16.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:29:16.860 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:16.861+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:16.916 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:16.916 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:16.924 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone38 from 0xaa9b800 (overlap 0x2788a00) to 0x17b8600 2024-05-06T00:29:16.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:16.977 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:16.978+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:17.054 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:17.054+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:17.068 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:17.068 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:17.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:17.155 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:17.153+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:17.250 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:17.250 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:17.255 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:17.254+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:17.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:17.356 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:17.354+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:17.422 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:17.423 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:17.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:17.481 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:17.482+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:17.558 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:17.557+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:17.602 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:17.603 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:17.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:17.659 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:17.658+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:17.752 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:17.752 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:17.761 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:17.762+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:17.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:17.777 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:29:17.778 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:29:17.778 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-06T00:29:17.778 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.272721 2024-05-06T00:29:17.778 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0.2727207526445484 2024-05-06T00:29:17.797 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #37, image image_client.0-clone37 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent38 2024-05-06T00:29:17.862 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:17.862+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:17.893 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:17.893 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:17.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:17.962 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:17.961+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:18.001 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:18.035 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:18.036 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:18.052 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:18.063 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:18.062+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:18.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:18.143 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:18.164 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:18.162+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:18.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:18.192 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:18.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:18.227 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:18.228 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:18.242 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:18.258 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:18.259 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:18.265 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:18.266+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:18.361 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:18.361 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:18.361 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:18.361 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:18.361 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:18.365 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:18.365+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:18.375 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:18.466 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:18.465+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:18.544 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:18.544 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:18.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:18.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:18.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.51701989, 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:29:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:29:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516999100999996, 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:29:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:29:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516979039000006, 2024-05-06T00:29:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:29:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:29:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.557 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516959763000003, 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:18.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:29:18.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516939644999994, 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.562 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:18.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.566 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516908236000006, 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.570 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516875323999997, 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:29:18.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:29:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:29:18.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516854737000003, 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:18.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:18.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:29:18.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516835357000005, 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:29:18.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:29:18.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516790675999999, 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:29:18.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:29:18.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:29:18.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:29:18.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:29:18.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516725585000003, 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:29:18.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:29:18.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516694010999998, 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:29:18.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:18.598 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.51667175, 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:18.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:29:18.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.516650634000001, 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.603 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:29:18.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:18.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:29:18.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.858934008000006, 2024-05-06T00:29:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:29:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:29:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:29:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:29:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:29:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:29:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:29:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:29:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:29:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:29:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:29:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:29:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:29:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:29:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:29:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:29:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.858523766000005, 2024-05-06T00:29:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:29:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:29:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:29:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:29:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:29:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:29:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:29:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:29:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:29:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:29:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:29:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:29:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:29:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:29:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:29:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:29:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.919879172000002, 2024-05-06T00:29:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:29:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:29:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:29:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:29:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:29:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:29:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:29:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:29:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:29:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:29:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:29:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:29:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:29:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:29:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:29:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:29:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:29:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:29:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:29:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:29:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.919807888000001, 2024-05-06T00:29:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:29:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:29:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:29:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:29:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:29:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:29:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:29:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:29:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:29:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:29:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:29:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:29:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:29:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:29:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:29:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:29:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.919776254999999, 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:29:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:29:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.915557709000002, 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:29:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:29:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:29:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:18.664 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:18.665 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:18.666 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:18.665+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:18.767 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:18.765+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:18.790 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.2 primary-affinity to 0.272721 (8178732) 2024-05-06T00:29:18.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:29:18.850 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:18.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:18.850 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:18.850 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:18.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:18.867 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:18.866+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:18.969 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:18.969+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:19.012 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:19.012 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:19.012 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:19.012 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:19.012 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:19.024 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:19.070 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:19.069+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:19.171 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:19.170+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:19.191 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:19.191 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:19.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:19.191 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:19.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.191 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 111.184181616, 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 111.18395304400001, 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 111.18387576000001, 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:19.205 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 111.183679779, 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:29:19.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:29:19.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.276570402999994, 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.210 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:19.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.214 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.276509524000005, 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.158953073999996, 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:29:19.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:29:19.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:29:19.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:29:19.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:29:19.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:29:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:29:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:29:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:29:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:29:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:29:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.158908346000004, 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.226 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:19.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:29:19.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:29:19.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:29:19.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.158869945000006, 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.231 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:29:19.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.235 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.158832683, 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:29:19.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.158795773999998, 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.244 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.158757855999994, 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:29:19.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:29:19.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:29:19.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.158720740999996, 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:29:19.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.152940291999997, 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.254 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.152902208, 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:29:19.258 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:29:19.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.152864534000003, 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:29:19.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:29:19.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.143429024, 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:19.267 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:29:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.143396597000006, 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.143372714999998, 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:29:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.143352785000005, 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:19.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:19.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:19.283 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:19.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:19.285 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:19.285+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:19.385 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:19.386+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3.66b1028d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.60249276799999996, 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60254045099999998, 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for peered", 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:29:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3.66b1028d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.60239229599999999, 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60257875100000002, 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for peered", 2024-05-06T00:29:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 2 2024-05-06T00:29:19.424 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:19.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:19.517 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:19.518+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:19.586 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:19.586+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:19.594 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:19.594 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:19.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:19.693 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:19.694+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:19.774 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.16934905599999, 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:29:19.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:29:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:29:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:594 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.896700357, 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034901626999999998, 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 594 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875048+0000", 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875049+0000", 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2699999999999999e-07 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875052+0000", 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9570000000000004e-06 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875062+0000", 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5389999999999996e-06 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875069+0000", 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4630000000000004e-06 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904314+0000", 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029244453 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.904476+0000", 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016184200000000001 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.905229+0000", 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075350799999999996 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908442+0000", 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032127050000000002 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909716+0000", 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012741320000000001 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909901+0000", 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185095 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909949+0000", 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8205999999999999e-05 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.896665181, 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:29:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:29:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.896576502, 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:29:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.896474559, 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:29:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:29:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:29:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.896365457, 2024-05-06T00:29:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:29:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:29:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.89626759799999, 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:29:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:29:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:29:19.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.89620254800001, 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:29:19.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.89610990200001, 2024-05-06T00:29:19.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:29:19.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:29:19.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.896037898, 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:29:19.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:29:19.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.838 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.839 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:19.839 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.136576412, 2024-05-06T00:29:19.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:29:19.839 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.839 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.839 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:29:19.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.840 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:19.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:19.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:19.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:29:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:29:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:29:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.847 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.136557164, 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:19.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:29:19.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.136507044, 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:29:19.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:29:19.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:29:19.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.13648599299999, 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:29:19.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:19.865 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.13647067399999, 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:29:19.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:29:19.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.136453792, 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:29:19.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.136435456, 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:29:19.878 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:29:19.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:29:19.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.13641945400001, 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:29:19.884 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:29:19.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:29:19.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:29:19.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.136333091, 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.890 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:19.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:29:19.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:29:19.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3331 3.d 3:b4d96372:::rbd_data.110cf57807c6.00000000000000ee:head [set-alloc-hint object_size 1048576 write_size 1048576,write 249856~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.126370615, 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034980070000000002, 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:19.896 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645378+0000", 2024-05-06T00:29:19.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645379+0000", 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.269e-06 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645391+0000", 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2123000000000001e-05 2024-05-06T00:29:19.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645406+0000", 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5339999999999999e-05 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.645415+0000", 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8990000000000007e-06 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675236+0000", 2024-05-06T00:29:19.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029821062999999998 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675301+0000", 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4935000000000001e-05 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675421+0000", 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119799 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677054+0000", 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001633172 2024-05-06T00:29:19.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680237+0000", 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031824129999999998 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680285+0000", 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8418999999999998e-05 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:19.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680358+0000", 2024-05-06T00:29:19.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2638000000000005e-05 2024-05-06T00:29:19.902 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.902 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.902 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.902 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:19.902 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:19.902 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:19.904 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:29:19.904 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:19.905 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:19.901+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:20.005 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:20.006+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:20.084 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:20.084 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:20.085 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:20.085 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:20.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:20.106 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:20.105+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:20.206 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:20.205+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:20.257 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:20.257 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:20.258 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:20.258 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:20.258 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:20.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:20.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:20.317 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:20.318+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:20.345 INFO:teuthology.orchestra.run.smithi186.stdout:5174 punch from 0x8a90200 to 0x8a91400, (0x1200 bytes) 2024-05-06T00:29:20.361 INFO:teuthology.orchestra.run.smithi186.stdout:5176 write 0x4d39200 thru 0x4d46d39 (0xdb3a bytes) 2024-05-06T00:29:20.361 INFO:teuthology.orchestra.run.smithi186.stdout:5177 read 0x25a1000 thru 0x25af421 (0xe422 bytes) 2024-05-06T00:29:20.363 INFO:teuthology.orchestra.run.smithi186.stdout:5178 read 0xa8cb600 thru 0xa8d6b01 (0xb502 bytes) 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.738169068999994, 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758927564000004, 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758853345000006, 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758791446000004, 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758731847000007, 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758647711999998, 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758582199000003, 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758524336999997, 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758427741999995, 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758350016999998, 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758277230999994, 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758199739999995, 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758105611000005, 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.758022648999997, 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.757948544000001, 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.757849355000005, 2024-05-06T00:29:20.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:20.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.757786202999995, 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:20.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.757685937999995, 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.482 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:20.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.486 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.757596566000004, 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.511737882999995, 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:20.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:20.495 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:20.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:20.497 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:20.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:20.498 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:20.498 INFO:teuthology.orchestra.run.smithi186.stdout:5179 trunc from 0xaa9b800 to 0x4175600 2024-05-06T00:29:20.498 INFO:teuthology.orchestra.run.smithi186.stdout:5180 read 0x352ce00 thru 0x352f5fe (0x27ff bytes) 2024-05-06T00:29:20.499 INFO:teuthology.orchestra.run.smithi186.stdout:5187 write 0x9de6200 thru 0x9deee40 (0x8c41 bytes) 2024-05-06T00:29:20.499 INFO:teuthology.orchestra.run.smithi186.stdout:5189 read 0x8264800 thru 0x82669fa (0x21fb bytes) 2024-05-06T00:29:20.499 INFO:teuthology.orchestra.run.smithi186.stdout:5190 punch from 0x2d56400 to 0x2d63c00, (0xd800 bytes) 2024-05-06T00:29:20.499 INFO:teuthology.orchestra.run.smithi186.stdout:5191 trunc from 0x9def000 to 0x5766c00 2024-05-06T00:29:20.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:20.499 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:20.498+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:20.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:20.500 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:20.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:20.500 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:20.519 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:20.519 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:20.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:20.540 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:20.572 INFO:teuthology.orchestra.run.smithi186.stdout:5192 write 0xd167a00 thru 0xd16a3a1 (0x29a2 bytes) 2024-05-06T00:29:20.574 INFO:teuthology.orchestra.run.smithi186.stdout:5193 read 0x64bf400 thru 0x64c2d9b (0x399c bytes) 2024-05-06T00:29:20.574 INFO:teuthology.orchestra.run.smithi186.stdout:5197 read 0x2f62200 thru 0x2f6fccb (0xdacc bytes) 2024-05-06T00:29:20.598 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:20.597+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:20.610 INFO:teuthology.orchestra.run.smithi186.stdout:5199 write 0xed2ac00 thru 0xed2faa7 (0x4ea8 bytes) 2024-05-06T00:29:20.612 INFO:teuthology.orchestra.run.smithi186.stdout:5201 write 0x5612400 thru 0x56164fb (0x40fc bytes) 2024-05-06T00:29:20.627 INFO:teuthology.orchestra.run.smithi186.stdout:5203 trunc from 0xed2fc00 to 0x7d08e00 2024-05-06T00:29:20.670 INFO:teuthology.orchestra.run.smithi186.stdout:5204 read 0xaa3e00 thru 0xaa68a9 (0x2aaa bytes) 2024-05-06T00:29:20.671 INFO:teuthology.orchestra.run.smithi186.stdout:5205 read 0x1c24400 thru 0x1c3323e (0xee3f bytes) 2024-05-06T00:29:20.673 INFO:teuthology.orchestra.run.smithi186.stdout:5208 read 0x1c3d600 thru 0x1c3e2a6 (0xca7 bytes) 2024-05-06T00:29:20.683 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:20.683 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:20.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:20.699 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:20.697+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:20.715 INFO:teuthology.orchestra.run.smithi186.stdout:5209 write 0xdfa6a00 thru 0xdfab4cc (0x4acd bytes) 2024-05-06T00:29:20.717 INFO:teuthology.orchestra.run.smithi186.stdout:5212 write 0x55e8200 thru 0x55eb48d (0x328e bytes) 2024-05-06T00:29:20.717 INFO:teuthology.orchestra.run.smithi186.stdout:5219 read 0x13f0200 thru 0x13f81b6 (0x7fb7 bytes) 2024-05-06T00:29:20.718 INFO:teuthology.orchestra.run.smithi186.stdout:5220 read 0x95f6c00 thru 0x95fd3f4 (0x67f5 bytes) 2024-05-06T00:29:20.733 INFO:teuthology.orchestra.run.smithi186.stdout:5221 trunc from 0xdfab600 to 0x83b4800 2024-05-06T00:29:20.777 INFO:teuthology.orchestra.run.smithi186.stdout:5223 write 0x1adac00 thru 0x1ae1159 (0x655a bytes) 2024-05-06T00:29:20.800 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:20.801+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:20.825 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:20.825 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:20.829 INFO:teuthology.orchestra.run.smithi186.stdout:5224 write 0xec0bc00 thru 0xec0c1c0 (0x5c1 bytes) 2024-05-06T00:29:20.831 INFO:teuthology.orchestra.run.smithi186.stdout:5225 write 0x58d1600 thru 0x58e09eb (0xf3ec bytes) 2024-05-06T00:29:20.831 INFO:teuthology.orchestra.run.smithi186.stdout:5226 write 0x4512c00 thru 0x4516655 (0x3a56 bytes) 2024-05-06T00:29:20.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:20.841 INFO:teuthology.orchestra.run.smithi186.stdout:5229 trunc from 0xec0c200 to 0x9afb600 2024-05-06T00:29:20.893 INFO:teuthology.orchestra.run.smithi186.stdout:5231 read 0x99b4200 thru 0x99b65a0 (0x23a1 bytes) 2024-05-06T00:29:20.895 INFO:teuthology.orchestra.run.smithi186.stdout:5232 punch from 0x6c90400 to 0x6c9b200, (0xae00 bytes) 2024-05-06T00:29:20.895 INFO:teuthology.orchestra.run.smithi186.stdout:5233 write 0x4a72a00 thru 0x4a76774 (0x3d75 bytes) 2024-05-06T00:29:20.896 INFO:teuthology.orchestra.run.smithi186.stdout:5234 punch from 0x88000 to 0x8ca00, (0x4a00 bytes) 2024-05-06T00:29:20.896 INFO:teuthology.orchestra.run.smithi186.stdout:5235 read 0x9aba00 thru 0x9bb3fb (0xf9fc bytes) 2024-05-06T00:29:20.898 INFO:teuthology.orchestra.run.smithi186.stdout:5237 write 0x738a000 thru 0x738d123 (0x3124 bytes) 2024-05-06T00:29:20.898 INFO:teuthology.orchestra.run.smithi186.stdout:5240 read 0x8d90800 thru 0x8d96cda (0x64db bytes) 2024-05-06T00:29:20.900 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:20.902+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:20.930 INFO:teuthology.orchestra.run.smithi186.stdout:5241 write 0x9f6f200 thru 0x9f70a3c (0x183d bytes) 2024-05-06T00:29:20.933 INFO:teuthology.orchestra.run.smithi186.stdout:5242 read 0x54aa400 thru 0x54b16f6 (0x72f7 bytes) 2024-05-06T00:29:20.933 INFO:teuthology.orchestra.run.smithi186.stdout:5244 read 0x5dc5a00 thru 0x5dc93f8 (0x39f9 bytes) 2024-05-06T00:29:20.934 INFO:teuthology.orchestra.run.smithi186.stdout:5246 write 0x4a8d400 thru 0x4a97d40 (0xa941 bytes) 2024-05-06T00:29:20.934 INFO:teuthology.orchestra.run.smithi186.stdout:5247 clone 40 order 21 su 16384 sc 5 2024-05-06T00:29:20.962 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:20.962 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:20.976 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:21.024 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:21.025+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:21.103 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:21.101+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:21.105 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:21.105 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:21.119 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:21.204 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:21.205+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:21.283 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:21.283 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:21.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:21.305 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:21.306+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:21.407 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:21.406+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:21.465 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:21.466 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:21.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:21.528 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:21.529+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:21.602 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:21.602 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:21.608 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:21.606+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:21.617 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:21.709 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:21.710+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:21.776 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:21.776 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:21.792 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:21.809 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:21.810+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:21.905 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:29:21.917 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:21.917 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:21.917 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:21.917+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:21.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:21.981 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone39 from 0x9f70c00 (overlap 0x4175600) to 0x20baa00 2024-05-06T00:29:22.021 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:22.021+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:22.111 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:22.110+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:22.141 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:22.141 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:22.155 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:22.244 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:22.245+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:22.313 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:22.313+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:22.318 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:22.318 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:22.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:22.414 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:22.414+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:22.498 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:22.499 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:22.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:22.515 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:22.514+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:22.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:22.616 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:22.614+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:22.635 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #38, image image_client.0-clone38 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent39 2024-05-06T00:29:22.646 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:22.647 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:22.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:22.695 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:22.695 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:22.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:22.712 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:22.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:22.749 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:22.750+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:22.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:22.750 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:22.770 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:22.771 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:22.785 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:22.786 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:22.816 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:22.817+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:29:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:22.855 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:22.918 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:22.917+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:22.985 INFO:teuthology.orchestra.run.smithi186.stdout:5249 trunc from 0x9f70c00 to 0x735400 2024-05-06T00:29:23.022 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:23.017+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 173.43045963899999, 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:29:23.035 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:29:23.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:29:23.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 125.15777576399999, 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:29:23.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 125.15768708500001, 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.049 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:29:23.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:29:23.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:29:23.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:29:23.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:29:23.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:29:23.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:29:23.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:29:23.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:29:23.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:29:23.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:29:23.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:29:23.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:29:23.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:29:23.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:29:23.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:29:23.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.061 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:23.062 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:23.062 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 125.157585142, 2024-05-06T00:29:23.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:29:23.062 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.062 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.062 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:23.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 125.15747604000001, 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.069 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:29:23.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:29:23.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:29:23.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:29:23.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:29:23.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:29:23.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:29:23.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:29:23.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:29:23.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:29:23.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:29:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:29:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:29:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:29:23.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:29:23.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:29:23.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:29:23.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:29:23.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.078 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.079 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:23.079 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:23.080 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 125.157378181, 2024-05-06T00:29:23.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:29:23.080 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.081 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.081 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.081 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.083 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.083 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:29:23.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.084 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:23.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:23.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:29:23.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:29:23.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:29:23.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:29:23.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:29:23.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:29:23.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:29:23.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:29:23.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:29:23.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:29:23.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:29:23.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:29:23.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:29:23.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:29:23.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:29:23.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:29:23.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:29:23.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:29:23.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:29:23.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:29:23.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:23.107 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 125.157313131, 2024-05-06T00:29:23.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:29:23.108 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.109 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:29:23.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.109 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:23.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:23.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:29:23.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:29:23.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:29:23.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:23.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:29:23.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:29:23.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:29:23.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:29:23.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:29:23.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:29:23.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:29:23.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 125.157220485, 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:29:23.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:29:23.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:29:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 125.15714848099999, 2024-05-06T00:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.397686995, 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.397667747, 2024-05-06T00:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.397617627, 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:29:23.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.397596576, 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.146 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:29:23.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.397581257, 2024-05-06T00:29:23.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.153 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:23.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:23.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.159 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.397564375, 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:29:23.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:29:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:29:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.39754603900001, 2024-05-06T00:29:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:29:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.168 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:29:23.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.169 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:29:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.397530037, 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 124.397443674, 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.180 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:29:23.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [read 12582912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2181934280000002, 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857773900000002, 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:23.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:29:23.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:29:23.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.812763+0000", 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99789039000000002 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813015+0000", 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025184099999999999 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813243+0000", 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022833400000000001 2024-05-06T00:29:23.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [sparse-read 12587008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2180929559999996, 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99922042799999999, 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:23.191 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813844+0000", 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99888496199999999 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813901+0000", 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6561000000000003e-05 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813986+0000", 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5643999999999995e-05 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.196 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.196 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:23.197 INFO:teuthology.orchestra.run.smithi186.stdout:5250 write 0x433b800 thru 0x433fba2 (0x43a3 bytes) 2024-05-06T00:29:23.197 INFO:teuthology.orchestra.run.smithi186.stdout:5251 write 0x4e4d000 thru 0x4e56a76 (0x9a77 bytes) 2024-05-06T00:29:23.197 INFO:teuthology.orchestra.run.smithi186.stdout:5252 read 0x31c0e00 thru 0x31ca54e (0x974f bytes) 2024-05-06T00:29:23.197 INFO:teuthology.orchestra.run.smithi186.stdout:5253 trunc from 0x4e56c00 to 0xc6b3e00 2024-05-06T00:29:23.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:23.199 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:23.197+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:23.233 INFO:teuthology.orchestra.run.smithi186.stdout:5254 write 0x3920400 thru 0x392b762 (0xb363 bytes) 2024-05-06T00:29:23.234 INFO:teuthology.orchestra.run.smithi186.stdout:5255 write 0xb346e00 thru 0xb3538a3 (0xcaa4 bytes) 2024-05-06T00:29:23.254 INFO:teuthology.orchestra.run.smithi186.stdout:5260 trunc from 0xc6b3e00 to 0x463a000 2024-05-06T00:29:23.299 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:23.297+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:23.316 INFO:teuthology.orchestra.run.smithi186.stdout:5263 read 0x272dc00 thru 0x273c479 (0xe87a bytes) 2024-05-06T00:29:23.321 INFO:teuthology.orchestra.run.smithi186.stdout:5265 read 0x2c59200 thru 0x2c6787b (0xe67c bytes) 2024-05-06T00:29:23.335 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:23.335 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:23.335 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:23.335 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:23.341 INFO:teuthology.orchestra.run.smithi186.stdout:5266 trunc from 0x463a000 to 0xc534200 2024-05-06T00:29:23.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:23.363 INFO:teuthology.orchestra.run.smithi186.stdout:5270 read 0x7ef4400 thru 0x7efb871 (0x7472 bytes) 2024-05-06T00:29:23.364 INFO:teuthology.orchestra.run.smithi186.stdout:5275 read 0x17c3a00 thru 0x17d07ff (0xce00 bytes) 2024-05-06T00:29:23.369 INFO:teuthology.orchestra.run.smithi186.stdout:5277 trunc from 0xc534200 to 0x9cfa800 2024-05-06T00:29:23.393 INFO:teuthology.orchestra.run.smithi186.stdout:5278 punch from 0x3788400 to 0x3792c00, (0xa800 bytes) 2024-05-06T00:29:23.395 INFO:teuthology.orchestra.run.smithi186.stdout:5280 read 0x9332a00 thru 0x933f804 (0xce05 bytes) 2024-05-06T00:29:23.395 INFO:teuthology.orchestra.run.smithi186.stdout:5281 punch from 0x6f61c00 to 0x6f64600, (0x2a00 bytes) 2024-05-06T00:29:23.395 INFO:teuthology.orchestra.run.smithi186.stdout:5282 punch from 0x7cf4800 to 0x7cfac00, (0x6400 bytes) 2024-05-06T00:29:23.399 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:23.398+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:23.405 INFO:teuthology.orchestra.run.smithi186.stdout:5283 trunc from 0x9cfa800 to 0x61d9e00 2024-05-06T00:29:23.450 INFO:teuthology.orchestra.run.smithi186.stdout:5286 trunc from 0x61d9e00 to 0x5c05e00 2024-05-06T00:29:23.474 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:23.474 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:23.475 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:23.475 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:23.475 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:23.482 INFO:teuthology.orchestra.run.smithi186.stdout:5287 punch from 0x4fa2e00 to 0x4fafe00, (0xd000 bytes) 2024-05-06T00:29:23.483 INFO:teuthology.orchestra.run.smithi186.stdout:5289 write 0x2276e00 thru 0x22829a3 (0xbba4 bytes) 2024-05-06T00:29:23.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:23.537 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:23.538+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:23.541 INFO:teuthology.orchestra.run.smithi186.stdout:5290 write 0xeab5400 thru 0xeac0768 (0xb369 bytes) 2024-05-06T00:29:23.542 INFO:teuthology.orchestra.run.smithi186.stdout:5291 read 0xc4ce600 thru 0xc4d0dbb (0x27bc bytes) 2024-05-06T00:29:23.543 INFO:teuthology.orchestra.run.smithi186.stdout:5292 write 0x28f5600 thru 0x2900ec6 (0xb8c7 bytes) 2024-05-06T00:29:23.543 INFO:teuthology.orchestra.run.smithi186.stdout:5295 read 0x50fc800 thru 0x510c265 (0xfa66 bytes) 2024-05-06T00:29:23.543 INFO:teuthology.orchestra.run.smithi186.stdout:5296 punch from 0xe896200 to 0xe89c600, (0x6400 bytes) 2024-05-06T00:29:23.545 INFO:teuthology.orchestra.run.smithi186.stdout:5298 write 0x475e400 thru 0x4768fab (0xabac bytes) 2024-05-06T00:29:23.556 INFO:teuthology.orchestra.run.smithi186.stdout:5300 trunc from 0xeac0800 to 0x7a5fa00 2024-05-06T00:29:23.601 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:23.601+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:23.627 INFO:teuthology.orchestra.run.smithi186.stdout:5304 write 0xa742400 thru 0xa74865a (0x625b bytes) 2024-05-06T00:29:23.628 INFO:teuthology.orchestra.run.smithi186.stdout:5305 punch from 0x2a18000 to 0x2a1f800, (0x7800 bytes) 2024-05-06T00:29:23.629 INFO:teuthology.orchestra.run.smithi186.stdout:5306 punch from 0x33be000 to 0x33c8c00, (0xac00 bytes) 2024-05-06T00:29:23.629 INFO:teuthology.orchestra.run.smithi186.stdout:5307 punch from 0xa496400 to 0xa49b600, (0x5200 bytes) 2024-05-06T00:29:23.640 INFO:teuthology.orchestra.run.smithi186.stdout:5308 trunc from 0xa748800 to 0x54ff200 2024-05-06T00:29:23.666 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.005893824, 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.667 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:23.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:23.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:23.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026652318999993, 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.673 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026578099999995, 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.677 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:23.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026516201000007, 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.682 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:23.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:23.686 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026456601999996, 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:23.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:23.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026372467000002, 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.691 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:23.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026306954000006, 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.700 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026249092, 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:23.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026152496999998, 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:23.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:23.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026074772000001, 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.709 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:23.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:23.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.713 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.026001985999997, 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:23.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:23.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.025924494999998, 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:23.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:23.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:23.722 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.025830365999994, 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:23.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.025747404000001, 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.731 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.025673299000005, 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:23.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.025574109999994, 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:23.740 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.025510957999998, 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:23.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:23.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.025410692999998, 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:23.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:23.749 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.025321321000007, 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:23.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.779462637999998, 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.754 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:23.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:23.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.758 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:23.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:23.761 INFO:teuthology.orchestra.run.smithi186.stdout:5309 punch from 0x428b200 to 0x4295a00, (0xa800 bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5310 punch from 0x19ef400 to 0x19f4800, (0x5400 bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5312 write 0x3dfe200 thru 0x3dfe4bd (0x2be bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5313 punch from 0x4d24000 to 0x4d28400, (0x4400 bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5314 read 0x2b8a000 thru 0x2b97a48 (0xda49 bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5315 read 0xb37400 thru 0xb3ae18 (0x3a19 bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5316 write 0x7267c00 thru 0x7272281 (0xa682 bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5317 write 0x343c200 thru 0x343c7cf (0x5d0 bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5319 write 0x2cc1400 thru 0x2cc2685 (0x1286 bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5320 trunc from 0x7272400 to 0x6e3e600 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5324 read 0x13d7200 thru 0x13d96de (0x24df bytes) 2024-05-06T00:29:23.762 INFO:teuthology.orchestra.run.smithi186.stdout:5325 trunc from 0x6e3e600 to 0x5a63a00 2024-05-06T00:29:23.763 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:23.757+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:23.793 INFO:teuthology.orchestra.run.smithi186.stdout:5328 trunc from 0x5a63a00 to 0x9e31800 2024-05-06T00:29:23.818 INFO:teuthology.orchestra.run.smithi186.stdout:5331 trunc from 0x9e31800 to 0x6f88a00 2024-05-06T00:29:23.819 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:29:23.819 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:29:23.819 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-06T00:29:23.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-06T00:29:23.840 INFO:teuthology.orchestra.run.smithi186.stdout:5333 punch from 0x32cda00 to 0x32d4c00, (0x7200 bytes) 2024-05-06T00:29:23.842 INFO:teuthology.orchestra.run.smithi186.stdout:5335 punch from 0x505ba00 to 0x505be00, (0x400 bytes) 2024-05-06T00:29:23.842 INFO:teuthology.orchestra.run.smithi186.stdout:5336 read 0x1b6f400 thru 0x1b71afb (0x26fc bytes) 2024-05-06T00:29:23.849 INFO:teuthology.orchestra.run.smithi186.stdout:5337 trunc from 0x6f88a00 to 0x9795e00 2024-05-06T00:29:23.861 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:23.862+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:23.869 INFO:teuthology.orchestra.run.smithi186.stdout:5342 punch from 0x16ed400 to 0x16f0000, (0x2c00 bytes) 2024-05-06T00:29:23.870 INFO:teuthology.orchestra.run.smithi186.stdout:5343 write 0x8cc6c00 thru 0x8cc7815 (0xc16 bytes) 2024-05-06T00:29:23.894 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.6270.0:2172 3.10 e142/132 3:0ee3e6cd:::rbd_object_map.110c73ad67ca:head v 142'536, mlcod=142'534)", 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:23.894329+0000", 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0010270310000000001, 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010436849999999999, 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:23.894329+0000", 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:23.894329+0000", 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:23.894331+0000", 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7099999999999999e-06 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:23.894342+0000", 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1116e-05 2024-05-06T00:29:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:23.894368+0000", 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6253000000000001e-05 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:23.894380+0000", 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.237e-05 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:23.894475+0000", 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4775e-05 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:23.894510+0000", 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5027000000000002e-05 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-05-06T00:29:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-05-06T00:29:23.918 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:23.919 INFO:teuthology.orchestra.run.smithi186.stdout:5344 write 0xd43ea00 thru 0xd44e2fc (0xf8fd bytes) 2024-05-06T00:29:23.919 INFO:teuthology.orchestra.run.smithi186.stdout:5345 write 0x7e0e600 thru 0x7e1bde5 (0xd7e6 bytes) 2024-05-06T00:29:23.920 INFO:teuthology.orchestra.run.smithi186.stdout:5348 write 0x6c4a200 thru 0x6c55c82 (0xba83 bytes) 2024-05-06T00:29:23.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:23.921 INFO:teuthology.orchestra.run.smithi186.stdout:5351 trunc from 0xd44e400 to 0x5fd7e00 2024-05-06T00:29:23.965 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:23.966+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:23.982 INFO:teuthology.orchestra.run.smithi186.stdout:5354 trunc from 0x5fd7e00 to 0xa63ac00 2024-05-06T00:29:24.000 INFO:teuthology.orchestra.run.smithi186.stdout:5355 trunc from 0xa63ac00 to 0xc085400 2024-05-06T00:29:24.015 INFO:teuthology.orchestra.run.smithi186.stdout:5356 trunc from 0xc085400 to 0xe7bd600 2024-05-06T00:29:24.026 INFO:teuthology.orchestra.run.smithi186.stdout:5357 read 0x3b75800 thru 0x3b811af (0xb9b0 bytes) 2024-05-06T00:29:24.028 INFO:teuthology.orchestra.run.smithi186.stdout:5358 read 0xa9cd600 thru 0xa9cd769 (0x16a bytes) 2024-05-06T00:29:24.028 INFO:teuthology.orchestra.run.smithi186.stdout:5359 read 0x6fe4800 thru 0x6fe96a3 (0x4ea4 bytes) 2024-05-06T00:29:24.037 INFO:teuthology.orchestra.run.smithi186.stdout:5361 trunc from 0xe7bd600 to 0xb1f4800 2024-05-06T00:29:24.060 INFO:teuthology.orchestra.run.smithi186.stdout:5364 trunc from 0xb1f4800 to 0x7b70a00 2024-05-06T00:29:24.062 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:24.061+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:24.091 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:24.092 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:24.092 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:24.092 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:24.092 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:24.098 INFO:teuthology.orchestra.run.smithi186.stdout:5366 write 0x9550200 thru 0x9557ca7 (0x7aa8 bytes) 2024-05-06T00:29:24.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:24.116 INFO:teuthology.orchestra.run.smithi186.stdout:5367 trunc from 0x9557e00 to 0xd35800 2024-05-06T00:29:24.163 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:24.161+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 142, 2024-05-06T00:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b583c614-b2ca-4769-90c0-44726bf69b34", 2024-05-06T00:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-06T00:26:16.923985+0000", 2024-05-06T00:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-06T00:29:22.811978+0000", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-06T00:26:27.394947+0000", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-06T00:28:54.626879+0000", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5806080, 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "nodeep-scrub", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "noscrub", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-06T00:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 6, 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "luminous", 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-06T00:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:28.436494+0000", 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "14", 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 5, 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 5, 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.40000000596046448, 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.61445111036300659, 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.74000000953674316, 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.61000001430511475 2024-05-06T00:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:33.941780+0000", 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 3.119999885559082, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 3.119999885559082, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 1.2000000476837158, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 3.75, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 3.75, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.61445111036300659, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.74000000953674316, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.61000001430511475 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:42.622595+0000", 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "141", 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 43, 2024-05-06T00:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 141, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 5.5300002098083496, 2024-05-06T00:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 1.4099999666213989, 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 1.2000000476837158, 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 6.6399998664855957, 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.690000057220459, 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.61445111036300659, 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.74000000953674316, 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.61000001430511475, 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "error_message": "pool 3 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d31b9a6a-864b-4c24-99db-8cc6cf3a5c24", 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 139, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6817/1427564096", 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6819/1427564096", 2024-05-06T00:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6823/1427564096", 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6821/1427564096", 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "86509826-0e53-4678-8260-17c97c05af54", 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0.9229278564453125, 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 139, 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6803/2663808859", 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6807/2663808859", 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/2663808859", 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6811/2663808859", 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "8973b524-521e-41c0-87f2-85cb390fbfeb", 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.2727203369140625, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 127, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6802/385938386", 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6806/385938386", 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6814/385938386", 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6810/385938386", 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "3b0d07bd-132e-48ff-83ac-7182eee05df9", 2024-05-06T00:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 139, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6811", 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6812", 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6814", 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6816", 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6821", 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6822", 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6817", 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6819", 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6812/4178038317", 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6816/4178038317", 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6822/4178038317", 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6819/4178038317", 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2", 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.833282470703125, 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 132, 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6808", 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6809", 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6810", 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6813", 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6820", 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6823", 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6815", 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6818", 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6809/1052555350", 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6813/1052555350", 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1052555350", 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6818/1052555350", 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "34b61f5f-e407-4414-8bc5-11263bbe261a", 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.5807037353515625, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 135, 2024-05-06T00:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6800", 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6801", 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6802", 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6803", 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6806", 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6807", 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6804", 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6805", 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6801/2019146845", 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6803/2019146845", 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6807/2019146845", 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6805/2019146845", 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-06T00:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.276988+0000", 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.054473+0000", 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.154558+0000", 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.284746+0000", 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.259847+0000", 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.180375+0000", 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [ 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.15", 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: 4, 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [ 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.6", 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 1, 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 4 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [], 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [], 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [], 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:24.233 INFO:teuthology.orchestra.run.smithi186.stdout:5370 trunc from 0xd35800 to 0x27a3a00 2024-05-06T00:29:24.233 INFO:teuthology.orchestra.run.smithi186.stdout:5372 punch from 0x4ee000 to 0x4f0e00, (0x2e00 bytes) 2024-05-06T00:29:24.233 INFO:teuthology.orchestra.run.smithi186.stdout:5373 read 0x184ac00 thru 0x184f64d (0x4a4e bytes) 2024-05-06T00:29:24.234 INFO:tasks.thrashosds.thrasher:j is {'epoch': 142, 'fsid': 'b583c614-b2ca-4769-90c0-44726bf69b34', 'created': '2024-05-06T00:26:16.923985+0000', 'modified': '2024-05-06T00:29:22.811978+0000', 'last_up_change': '2024-05-06T00:26:27.394947+0000', 'last_in_change': '2024-05-06T00:28:54.626879+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T00:26:28.436494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5, 'score_stable': 5, 'optimal_score': 0.4000000059604645, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6144511103630066, 'average_primary_affinity': 0.7400000095367432, 'average_primary_affinity_weighted': 0.6100000143051147}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-05-06T00:26:33.941780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.119999885559082, 'score_stable': 3.119999885559082, 'optimal_score': 1.2000000476837158, 'raw_score_acting': 3.75, 'raw_score_stable': 3.75, 'primary_affinity_weighted': 0.6144511103630066, 'average_primary_affinity': 0.7400000095367432, 'average_primary_affinity_weighted': 0.6100000143051147}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-06T00:26:42.622595+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 43, 'snap_epoch': 141, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.53000020980835, 'score_stable': 1.409999966621399, 'optimal_score': 1.2000000476837158, 'raw_score_acting': 6.639999866485596, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.6144511103630066, 'average_primary_affinity': 0.7400000095367432, 'average_primary_affinity_weighted': 0.6100000143051147, 'error_message': 'pool 3 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': 'd31b9a6a-864b-4c24-99db-8cc6cf3a5c24', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 139, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6817', 'nonce': 1427564096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6818', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 1427564096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6822', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 1427564096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6821', 'nonce': 1427564096}]}, 'public_addr': '172.21.15.78:6817/1427564096', 'cluster_addr': '172.21.15.78:6819/1427564096', 'heartbeat_back_addr': '172.21.15.78:6823/1427564096', 'heartbeat_front_addr': '172.21.15.78:6821/1427564096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '86509826-0e53-4678-8260-17c97c05af54', 'up': 1, 'in': 1, 'weight': 0.9229278564453125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 139, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6801', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6803', 'nonce': 2663808859}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6805', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6807', 'nonce': 2663808859}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6813', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 2663808859}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6809', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6811', 'nonce': 2663808859}]}, 'public_addr': '172.21.15.78:6803/2663808859', 'cluster_addr': '172.21.15.78:6807/2663808859', 'heartbeat_back_addr': '172.21.15.78:6815/2663808859', 'heartbeat_front_addr': '172.21.15.78:6811/2663808859', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8973b524-521e-41c0-87f2-85cb390fbfeb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2727203369140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6802', 'nonce': 385938386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6804', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6806', 'nonce': 385938386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6812', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6814', 'nonce': 385938386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6808', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6810', 'nonce': 385938386}]}, 'public_addr': '172.21.15.78:6802/385938386', 'cluster_addr': '172.21.15.78:6806/385938386', 'heartbeat_back_addr': '172.21.15.78:6814/385938386', 'heartbeat_front_addr': '172.21.15.78:6810/385938386', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3b0d07bd-132e-48ff-83ac-7182eee05df9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 139, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6811', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6812', 'nonce': 4178038317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6816', 'nonce': 4178038317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6821', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6822', 'nonce': 4178038317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6817', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 4178038317}]}, 'public_addr': '172.21.15.146:6812/4178038317', 'cluster_addr': '172.21.15.146:6816/4178038317', 'heartbeat_back_addr': '172.21.15.146:6822/4178038317', 'heartbeat_front_addr': '172.21.15.146:6819/4178038317', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.833282470703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 1052555350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 1052555350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1052555350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6815', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6818', 'nonce': 1052555350}]}, 'public_addr': '172.21.15.146:6809/1052555350', 'cluster_addr': '172.21.15.146:6813/1052555350', 'heartbeat_back_addr': '172.21.15.146:6823/1052555350', 'heartbeat_front_addr': '172.21.15.146:6818/1052555350', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '34b61f5f-e407-4414-8bc5-11263bbe261a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5807037353515625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 2019146845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 2019146845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 2019146845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 2019146845}]}, 'public_addr': '172.21.15.146:6801/2019146845', 'cluster_addr': '172.21.15.146:6803/2019146845', 'heartbeat_back_addr': '172.21.15.146:6807/2019146845', 'heartbeat_front_addr': '172.21.15.146:6805/2019146845', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.276988+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.054473+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.154558+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.284746+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.259847+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.180375+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.15', 'osds': [4, 1]}], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 1, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-06T00:29:24.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:29:24.235 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:24.235 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:24.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:24.235 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:24.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.235 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.272276171999998, 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:24.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.271091556999998, 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:24.241 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.245 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.387670512, 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.246 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.387633299000001, 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:24.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.255 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.387544493, 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.256 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:29:24.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:29:24.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.260 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.387484301000001, 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:24.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.387429998, 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.387363358, 2024-05-06T00:29:24.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:29:24.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:29:24.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.387301086000001, 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:29:24.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.387242090000001, 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.380837369, 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:24.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:29:24.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:29:24.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:29:24.286 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.380788635, 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.287 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:24.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:29:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.380744383, 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:29:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.380673351, 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.380622536000001, 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.301 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.371883388000001, 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.306 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:29:24.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:29:24.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:24.310 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.371860374000001, 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:24.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.371838124, 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:29:24.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:24.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:29:24.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.371728274000001, 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.319 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:29:24.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.371664224, 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:24.324 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:29:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:29:24.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:24.328 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:24.331 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:24.330+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:24.331 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:24.332 INFO:teuthology.orchestra.run.smithi186.stdout:5374 write 0x79dde00 thru 0x79edc71 (0xfe72 bytes) 2024-05-06T00:29:24.332 INFO:teuthology.orchestra.run.smithi186.stdout:5377 punch from 0x4cac200 to 0x4cae600, (0x2400 bytes) 2024-05-06T00:29:24.332 INFO:teuthology.orchestra.run.smithi186.stdout:5381 punch from 0x45ec800 to 0x45f6800, (0xa000 bytes) 2024-05-06T00:29:24.332 INFO:teuthology.orchestra.run.smithi186.stdout:5383 write 0xd2b800 thru 0xd2f368 (0x3b69 bytes) 2024-05-06T00:29:24.332 INFO:teuthology.orchestra.run.smithi186.stdout:5386 trunc from 0x79ede00 to 0x48f7400 2024-05-06T00:29:24.332 INFO:teuthology.orchestra.run.smithi186.stdout:5387 read 0x1668400 thru 0x167667c (0xe27d bytes) 2024-05-06T00:29:24.332 INFO:teuthology.orchestra.run.smithi186.stdout:5388 read 0x4316e00 thru 0x431ee53 (0x8054 bytes) 2024-05-06T00:29:24.333 INFO:teuthology.orchestra.run.smithi186.stdout:5392 read 0x3aff200 thru 0x3b00be7 (0x19e8 bytes) 2024-05-06T00:29:24.333 INFO:teuthology.orchestra.run.smithi186.stdout:5393 write 0x409800 thru 0x40991c (0x11d bytes) 2024-05-06T00:29:24.333 INFO:teuthology.orchestra.run.smithi186.stdout:5395 read 0x13aba00 thru 0x13b52de (0x98df bytes) 2024-05-06T00:29:24.333 INFO:teuthology.orchestra.run.smithi186.stdout:5396 write 0xc00a00 thru 0xc028c7 (0x1ec8 bytes) 2024-05-06T00:29:24.333 INFO:teuthology.orchestra.run.smithi186.stdout:5397 write 0xdf7ac00 thru 0xdf7f7f5 (0x4bf6 bytes) 2024-05-06T00:29:24.333 INFO:teuthology.orchestra.run.smithi186.stdout:5398 trunc from 0xdf7f800 to 0xddd2a00 2024-05-06T00:29:24.352 INFO:teuthology.orchestra.run.smithi186.stdout:5400 write 0x67b8400 thru 0x67c816c (0xfd6d bytes) 2024-05-06T00:29:24.353 INFO:teuthology.orchestra.run.smithi186.stdout:5401 write 0x9e92e00 thru 0x9e953f6 (0x25f7 bytes) 2024-05-06T00:29:24.353 INFO:teuthology.orchestra.run.smithi186.stdout:5405 punch from 0x8b75c00 to 0x8b85600, (0xfa00 bytes) 2024-05-06T00:29:24.353 INFO:teuthology.orchestra.run.smithi186.stdout:5407 read 0x2805a00 thru 0x2813d52 (0xe353 bytes) 2024-05-06T00:29:24.359 INFO:teuthology.orchestra.run.smithi186.stdout:5409 trunc from 0xddd2a00 to 0xb408e00 2024-05-06T00:29:24.401 INFO:teuthology.orchestra.run.smithi186.stdout:5410 write 0x99a5a00 thru 0x99b4fcb (0xf5cc bytes) 2024-05-06T00:29:24.403 INFO:teuthology.orchestra.run.smithi186.stdout:5411 read 0xaabb000 thru 0xaac2b48 (0x7b49 bytes) 2024-05-06T00:29:24.403 INFO:teuthology.orchestra.run.smithi186.stdout:5414 read 0x15d8a00 thru 0x15dbecc (0x34cd bytes) 2024-05-06T00:29:24.405 INFO:teuthology.orchestra.run.smithi186.stdout:5416 trunc from 0xb408e00 to 0xc04c600 2024-05-06T00:29:24.430 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:24.429+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:24.453 INFO:teuthology.orchestra.run.smithi186.stdout:5423 write 0xec36200 thru 0xec418ea (0xb6eb bytes) 2024-05-06T00:29:24.456 INFO:teuthology.orchestra.run.smithi186.stdout:5424 clone 41 order 21 su 65536 sc 3 2024-05-06T00:29:24.477 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:24.477 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:24.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.477 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.6270.0:3515 3.c 3.2e7a5bac (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e142)", 2024-05-06T00:29:24.477 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:29:24.479661+0000", 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.0189999999999999e-05, 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6006999999999993e-05, 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.6270", 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/1548275711", 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 3515 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:24.479661+0000", 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:24.479661+0000", 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:24.479661+0000", 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6599999999999994e-07 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:24.479666+0000", 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9130000000000002e-06 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:24.479677+0000", 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0414000000000001e-05 2024-05-06T00:29:24.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:24.479682+0000", 2024-05-06T00:29:24.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6640000000000003e-06 2024-05-06T00:29:24.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:24.479725+0000", 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2372e-05 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 1 2024-05-06T00:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:24.496 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:24.541 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:24.541+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:24.633 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:24.634+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:24.658 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:24.659 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:24.659 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:24.659 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:24.659 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:24.662 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:29:24.663 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:29:24.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:24.680 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-05-06T00:29:24.438055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1251000133,"num_objects":2049,"num_object_clones":472,"num_object_copies":4098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":1005,"num_read":226703,"num_read_kb":718053,"num_write":15834,"num_write_kb":170411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5554,"num_bytes_recovered":74299906,"num_keys_recovered":904,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14629,"ondisk_log_size":14629,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":430972,"kb_used_data":124540,"kb_used_omap":9,"kb_used_meta":306358,"kb_avail":565800068,"statfs":{"total":579820584960,"available":579379269632,"internally_reserved":0,"allocated":127528960,"data_stored":149515155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":313711264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5844094,"num_objects":199,"num_object_clones":38,"num_object_copies":398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":146,"num_read":5068,"num_read_kb":17063,"num_write":406,"num_write_kb":2192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.647342"},"pg_stats":[{"pgid":"3.1f","version":"141'466","reported_seq":11276,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.091801+0000","last_change":"2024-05-06T00:28:31.628317+0000","last_active":"2024-05-06T00:29:22.091801+0000","last_peered":"2024-05-06T00:29:22.091801+0000","last_clean":"2024-05-06T00:29:22.091801+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:29:22.091801+0000","last_undegraded":"2024-05-06T00:29:22.091801+0000","last_fullsized":"2024-05-06T00:29:22.091801+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:04:27.435048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":28351623,"num_objects":63,"num_object_clones":14,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":32,"num_read":10250,"num_read_kb":37491,"num_write":406,"num_write_kb":2392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"141'266","reported_seq":9289,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.040017+0000","last_change":"2024-05-06T00:28:58.676392+0000","last_active":"2024-05-06T00:29:22.040017+0000","last_peered":"2024-05-06T00:29:22.040017+0000","last_clean":"2024-05-06T00:29:22.040017+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:29:22.040017+0000","last_undegraded":"2024-05-06T00:29:22.040017+0000","last_fullsized":"2024-05-06T00:29:22.040017+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:21:10.095620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":74290760,"num_objects":74,"num_object_clones":22,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":29,"num_read":8534,"num_read_kb":46824,"num_write":239,"num_write_kb":17457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"141'366","reported_seq":3245,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.377324+0000","last_change":"2024-05-06T00:29:13.787230+0000","last_active":"2024-05-06T00:29:22.377324+0000","last_peered":"2024-05-06T00:29:22.377324+0000","last_clean":"2024-05-06T00:29:22.377324+0000","last_became_active":"2024-05-06T00:29:13.784483+0000","last_became_peered":"2024-05-06T00:29:13.784483+0000","last_unstale":"2024-05-06T00:29:22.377324+0000","last_undegraded":"2024-05-06T00:29:22.377324+0000","last_fullsized":"2024-05-06T00:29:22.377324+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:01.766347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":47189556,"num_objects":58,"num_object_clones":13,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":36,"num_read":3123,"num_read_kb":11592,"num_write":342,"num_write_kb":2598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"141'692","reported_seq":8162,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.495725+0000","last_change":"2024-05-06T00:29:07.739802+0000","last_active":"2024-05-06T00:29:22.495725+0000","last_peered":"2024-05-06T00:29:22.495725+0000","last_clean":"2024-05-06T00:29:22.495725+0000","last_became_active":"2024-05-06T00:29:07.735705+0000","last_became_peered":"2024-05-06T00:29:07.735705+0000","last_unstale":"2024-05-06T00:29:22.495725+0000","last_undegraded":"2024-05-06T00:29:22.495725+0000","last_fullsized":"2024-05-06T00:29:22.495725+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:57:35.820711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":43869471,"num_objects":68,"num_object_clones":18,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":28,"num_read":10796,"num_read_kb":29098,"num_write":602,"num_write_kb":5097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"141'892","reported_seq":7680,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.463284+0000","last_change":"2024-05-06T00:28:05.824053+0000","last_active":"2024-05-06T00:29:22.463284+0000","last_peered":"2024-05-06T00:29:22.463284+0000","last_clean":"2024-05-06T00:29:22.463284+0000","last_became_active":"2024-05-06T00:28:05.820651+0000","last_became_peered":"2024-05-06T00:28:05.820651+0000","last_unstale":"2024-05-06T00:29:22.463284+0000","last_undegraded":"2024-05-06T00:29:22.463284+0000","last_fullsized":"2024-05-06T00:29:22.463284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":892,"log_dups_size":0,"ondisk_log_size":892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:19:57.765396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":60875901,"num_objects":94,"num_object_clones":27,"num_object_copies":188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":44,"num_read":10403,"num_read_kb":30761,"num_write":947,"num_write_kb":7915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"141'316","reported_seq":5786,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.480420+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:29:22.480420+0000","last_peered":"2024-05-06T00:29:22.480420+0000","last_clean":"2024-05-06T00:29:22.480420+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:29:22.480420+0000","last_undegraded":"2024-05-06T00:29:22.480420+0000","last_fullsized":"2024-05-06T00:29:22.480420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:33:45.956941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":35626727,"num_objects":56,"num_object_clones":10,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":27,"num_read":6068,"num_read_kb":22703,"num_write":349,"num_write_kb":13536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"141'578","reported_seq":3091,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.472170+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:29:22.472170+0000","last_peered":"2024-05-06T00:29:22.472170+0000","last_clean":"2024-05-06T00:29:22.472170+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:29:22.472170+0000","last_undegraded":"2024-05-06T00:29:22.472170+0000","last_fullsized":"2024-05-06T00:29:22.472170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:20:10.374486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":19870666,"num_objects":62,"num_object_clones":16,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":26,"num_read":4677,"num_read_kb":14881,"num_write":757,"num_write_kb":5512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'582","reported_seq":13611,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.192030+0000","last_change":"2024-05-06T00:29:19.817685+0000","last_active":"2024-05-06T00:29:22.192030+0000","last_peered":"2024-05-06T00:29:22.192030+0000","last_clean":"2024-05-06T00:29:22.192030+0000","last_became_active":"2024-05-06T00:29:19.816383+0000","last_became_peered":"2024-05-06T00:29:19.816383+0000","last_unstale":"2024-05-06T00:29:22.192030+0000","last_undegraded":"2024-05-06T00:29:22.192030+0000","last_fullsized":"2024-05-06T00:29:22.192030+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:31:24.551672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":29062709,"num_objects":65,"num_object_clones":19,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":30,"num_read":13311,"num_read_kb":24246,"num_write":416,"num_write_kb":3199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"141'277","reported_seq":4369,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.085365+0000","last_change":"2024-05-06T00:28:25.971398+0000","last_active":"2024-05-06T00:29:22.085365+0000","last_peered":"2024-05-06T00:29:22.085365+0000","last_clean":"2024-05-06T00:29:22.085365+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:29:22.085365+0000","last_undegraded":"2024-05-06T00:29:22.085365+0000","last_fullsized":"2024-05-06T00:29:22.085365+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:29:56.590206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":81067619,"num_objects":66,"num_object_clones":15,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":34,"num_read":3505,"num_read_kb":18360,"num_write":252,"num_write_kb":5016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"141'426","reported_seq":2394,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.038126+0000","last_change":"2024-05-06T00:28:31.621968+0000","last_active":"2024-05-06T00:29:22.038126+0000","last_peered":"2024-05-06T00:29:22.038126+0000","last_clean":"2024-05-06T00:29:22.038126+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:29:22.038126+0000","last_undegraded":"2024-05-06T00:29:22.038126+0000","last_fullsized":"2024-05-06T00:29:22.038126+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:47:46.244586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":52991079,"num_objects":63,"num_object_clones":18,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":32,"num_read":2590,"num_read_kb":14138,"num_write":556,"num_write_kb":1796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"141'220","reported_seq":5041,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.066093+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:29:22.066093+0000","last_peered":"2024-05-06T00:29:22.066093+0000","last_clean":"2024-05-06T00:29:22.066093+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:29:22.066093+0000","last_undegraded":"2024-05-06T00:29:22.066093+0000","last_fullsized":"2024-05-06T00:29:22.066093+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:05:55.430735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15049254,"num_objects":50,"num_object_clones":14,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":27,"num_read":4740,"num_read_kb":10015,"num_write":215,"num_write_kb":15202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"141'427","reported_seq":7052,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.518279+0000","last_change":"2024-05-06T00:28:31.621451+0000","last_active":"2024-05-06T00:29:22.518279+0000","last_peered":"2024-05-06T00:29:22.518279+0000","last_clean":"2024-05-06T00:29:22.518279+0000","last_became_active":"2024-05-06T00:28:31.620668+0000","last_became_peered":"2024-05-06T00:28:31.620668+0000","last_unstale":"2024-05-06T00:29:22.518279+0000","last_undegraded":"2024-05-06T00:29:22.518279+0000","last_fullsized":"2024-05-06T00:29:22.518279+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:34:00.969673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":28512144,"num_objects":52,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":26,"num_read":7353,"num_read_kb":20551,"num_write":587,"num_write_kb":2326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":209,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.824837+0000","last_change":"2024-05-06T00:28:27.989114+0000","last_active":"2024-05-06T00:29:21.824837+0000","last_peered":"2024-05-06T00:29:21.824837+0000","last_clean":"2024-05-06T00:29:21.824837+0000","last_became_active":"2024-05-06T00:28:27.988260+0000","last_became_peered":"2024-05-06T00:28:27.988260+0000","last_unstale":"2024-05-06T00:29:21.824837+0000","last_undegraded":"2024-05-06T00:29:21.824837+0000","last_fullsized":"2024-05-06T00:29:21.824837+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:19:42.564014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'808","reported_seq":6945,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.091935+0000","last_change":"2024-05-06T00:28:06.817938+0000","last_active":"2024-05-06T00:29:22.091935+0000","last_peered":"2024-05-06T00:29:22.091935+0000","last_clean":"2024-05-06T00:29:22.091935+0000","last_became_active":"2024-05-06T00:28:06.816588+0000","last_became_peered":"2024-05-06T00:28:06.816588+0000","last_unstale":"2024-05-06T00:29:22.091935+0000","last_undegraded":"2024-05-06T00:29:22.091935+0000","last_fullsized":"2024-05-06T00:29:22.091935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:15:28.866011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":76338822,"num_objects":59,"num_object_clones":12,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":24,"num_read":10164,"num_read_kb":26054,"num_write":1021,"num_write_kb":7204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":271,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.816092+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:29:21.816092+0000","last_peered":"2024-05-06T00:29:21.816092+0000","last_clean":"2024-05-06T00:29:21.816092+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:29:21.816092+0000","last_undegraded":"2024-05-06T00:29:21.816092+0000","last_fullsized":"2024-05-06T00:29:21.816092+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"141'631","reported_seq":6042,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.041017+0000","last_change":"2024-05-06T00:29:19.818357+0000","last_active":"2024-05-06T00:29:22.041017+0000","last_peered":"2024-05-06T00:29:22.041017+0000","last_clean":"2024-05-06T00:29:22.041017+0000","last_became_active":"2024-05-06T00:29:19.817230+0000","last_became_peered":"2024-05-06T00:29:19.817230+0000","last_unstale":"2024-05-06T00:29:22.041017+0000","last_undegraded":"2024-05-06T00:29:22.041017+0000","last_fullsized":"2024-05-06T00:29:22.041017+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:38:00.594597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":29721735,"num_objects":62,"num_object_clones":16,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":29,"num_read":8349,"num_read_kb":19239,"num_write":620,"num_write_kb":2685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":140,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.832921+0000","last_change":"2024-05-06T00:28:27.997043+0000","last_active":"2024-05-06T00:29:21.832921+0000","last_peered":"2024-05-06T00:29:21.832921+0000","last_clean":"2024-05-06T00:29:21.832921+0000","last_became_active":"2024-05-06T00:28:27.996051+0000","last_became_peered":"2024-05-06T00:28:27.996051+0000","last_unstale":"2024-05-06T00:29:21.832921+0000","last_undegraded":"2024-05-06T00:29:21.832921+0000","last_fullsized":"2024-05-06T00:29:21.832921+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:03:48.195523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"141'525","reported_seq":8744,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.060119+0000","last_change":"2024-05-06T00:28:59.669820+0000","last_active":"2024-05-06T00:29:22.060119+0000","last_peered":"2024-05-06T00:29:22.060119+0000","last_clean":"2024-05-06T00:29:22.060119+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:29:22.060119+0000","last_undegraded":"2024-05-06T00:29:22.060119+0000","last_fullsized":"2024-05-06T00:29:22.060119+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:47:50.300850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":51007617,"num_objects":77,"num_object_clones":15,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":37,"num_read":9519,"num_read_kb":30431,"num_write":603,"num_write_kb":4906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":271,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.816475+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:29:21.816475+0000","last_peered":"2024-05-06T00:29:21.816475+0000","last_clean":"2024-05-06T00:29:21.816475+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:29:21.816475+0000","last_undegraded":"2024-05-06T00:29:21.816475+0000","last_fullsized":"2024-05-06T00:29:21.816475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"141'294","reported_seq":2775,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.083321+0000","last_change":"2024-05-06T00:28:58.680125+0000","last_active":"2024-05-06T00:29:22.083321+0000","last_peered":"2024-05-06T00:29:22.083321+0000","last_clean":"2024-05-06T00:29:22.083321+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:29:22.083321+0000","last_undegraded":"2024-05-06T00:29:22.083321+0000","last_fullsized":"2024-05-06T00:29:22.083321+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:47:59.261099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":20618875,"num_objects":43,"num_object_clones":8,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":21,"num_read":2720,"num_read_kb":9908,"num_write":307,"num_write_kb":1682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"141'269","reported_seq":2903,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.182816+0000","last_change":"2024-05-06T00:28:02.884939+0000","last_active":"2024-05-06T00:29:22.182816+0000","last_peered":"2024-05-06T00:29:22.182816+0000","last_clean":"2024-05-06T00:29:22.182816+0000","last_became_active":"2024-05-06T00:28:02.782051+0000","last_became_peered":"2024-05-06T00:28:02.782051+0000","last_unstale":"2024-05-06T00:29:22.182816+0000","last_undegraded":"2024-05-06T00:29:22.182816+0000","last_fullsized":"2024-05-06T00:29:22.182816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:42:50.464060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":27404976,"num_objects":58,"num_object_clones":14,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":31,"num_read":2642,"num_read_kb":15403,"num_write":307,"num_write_kb":1882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":116,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.816316+0000","last_change":"2024-05-06T00:29:07.737006+0000","last_active":"2024-05-06T00:29:21.816316+0000","last_peered":"2024-05-06T00:29:21.816316+0000","last_clean":"2024-05-06T00:29:21.816316+0000","last_became_active":"2024-05-06T00:29:07.736607+0000","last_became_peered":"2024-05-06T00:29:07.736607+0000","last_unstale":"2024-05-06T00:29:21.816316+0000","last_undegraded":"2024-05-06T00:29:21.816316+0000","last_fullsized":"2024-05-06T00:29:21.816316+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:33:39.712007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"141'513","reported_seq":5780,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.482995+0000","last_change":"2024-05-06T00:28:59.669906+0000","last_active":"2024-05-06T00:29:22.482995+0000","last_peered":"2024-05-06T00:29:22.482995+0000","last_clean":"2024-05-06T00:29:22.482995+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:29:22.482995+0000","last_undegraded":"2024-05-06T00:29:22.482995+0000","last_fullsized":"2024-05-06T00:29:22.482995+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:30:44.513069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":42837679,"num_objects":61,"num_object_clones":12,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":27,"num_read":5990,"num_read_kb":19817,"num_write":661,"num_write_kb":3930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.816164+0000","last_change":"2024-05-06T00:29:19.816862+0000","last_active":"2024-05-06T00:29:21.816164+0000","last_peered":"2024-05-06T00:29:21.816164+0000","last_clean":"2024-05-06T00:29:21.816164+0000","last_became_active":"2024-05-06T00:29:19.816456+0000","last_became_peered":"2024-05-06T00:29:19.816456+0000","last_unstale":"2024-05-06T00:29:21.816164+0000","last_undegraded":"2024-05-06T00:29:21.816164+0000","last_fullsized":"2024-05-06T00:29:21.816164+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:42:56.863932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"141'282","reported_seq":4418,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.126336+0000","last_change":"2024-05-06T00:28:05.821874+0000","last_active":"2024-05-06T00:29:22.126336+0000","last_peered":"2024-05-06T00:29:22.126336+0000","last_clean":"2024-05-06T00:29:22.126336+0000","last_became_active":"2024-05-06T00:28:05.820367+0000","last_became_peered":"2024-05-06T00:28:05.820367+0000","last_unstale":"2024-05-06T00:29:22.126336+0000","last_undegraded":"2024-05-06T00:29:22.126336+0000","last_fullsized":"2024-05-06T00:29:22.126336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:01:03.938559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":44108334,"num_objects":63,"num_object_clones":15,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":28,"num_read":3896,"num_read_kb":12743,"num_write":300,"num_write_kb":2174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":210,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.816292+0000","last_change":"2024-05-06T00:28:27.993179+0000","last_active":"2024-05-06T00:29:21.816292+0000","last_peered":"2024-05-06T00:29:21.816292+0000","last_clean":"2024-05-06T00:29:21.816292+0000","last_became_active":"2024-05-06T00:28:27.991961+0000","last_became_peered":"2024-05-06T00:28:27.991961+0000","last_unstale":"2024-05-06T00:29:21.816292+0000","last_undegraded":"2024-05-06T00:29:21.816292+0000","last_fullsized":"2024-05-06T00:29:21.816292+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:21:51.936053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"141'379","reported_seq":9328,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.506281+0000","last_change":"2024-05-06T00:28:25.970874+0000","last_active":"2024-05-06T00:29:22.506281+0000","last_peered":"2024-05-06T00:29:22.506281+0000","last_clean":"2024-05-06T00:29:22.506281+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:29:22.506281+0000","last_undegraded":"2024-05-06T00:29:22.506281+0000","last_fullsized":"2024-05-06T00:29:22.506281+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:44:54.587580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":20108953,"num_objects":50,"num_object_clones":10,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":25,"num_read":9715,"num_read_kb":22609,"num_write":301,"num_write_kb":2519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":108,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.820974+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:29:21.820974+0000","last_peered":"2024-05-06T00:29:21.820974+0000","last_clean":"2024-05-06T00:29:21.820974+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:29:21.820974+0000","last_undegraded":"2024-05-06T00:29:21.820974+0000","last_fullsized":"2024-05-06T00:29:21.820974+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":158,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:21.832665+0000","last_change":"2024-05-06T00:28:28.005137+0000","last_active":"2024-05-06T00:29:21.832665+0000","last_peered":"2024-05-06T00:29:21.832665+0000","last_clean":"2024-05-06T00:29:21.832665+0000","last_became_active":"2024-05-06T00:28:27.995351+0000","last_became_peered":"2024-05-06T00:28:27.995351+0000","last_unstale":"2024-05-06T00:29:21.832665+0000","last_undegraded":"2024-05-06T00:29:21.832665+0000","last_fullsized":"2024-05-06T00:29:21.832665+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:17:37.661069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"141'413","reported_seq":3378,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.068885+0000","last_change":"2024-05-06T00:29:19.819457+0000","last_active":"2024-05-06T00:29:22.068885+0000","last_peered":"2024-05-06T00:29:22.068885+0000","last_clean":"2024-05-06T00:29:22.068885+0000","last_became_active":"2024-05-06T00:29:19.818623+0000","last_became_peered":"2024-05-06T00:29:19.818623+0000","last_unstale":"2024-05-06T00:29:22.068885+0000","last_undegraded":"2024-05-06T00:29:22.068885+0000","last_fullsized":"2024-05-06T00:29:22.068885+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:26:27.262932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":28097766,"num_objects":70,"num_object_clones":16,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":39,"num_read":3610,"num_read_kb":11711,"num_write":472,"num_write_kb":2503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"141'471","reported_seq":5338,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.520761+0000","last_change":"2024-05-06T00:28:58.674634+0000","last_active":"2024-05-06T00:29:22.520761+0000","last_peered":"2024-05-06T00:29:22.520761+0000","last_clean":"2024-05-06T00:29:22.520761+0000","last_became_active":"2024-05-06T00:28:58.673289+0000","last_became_peered":"2024-05-06T00:28:58.673289+0000","last_unstale":"2024-05-06T00:29:22.520761+0000","last_undegraded":"2024-05-06T00:29:22.520761+0000","last_fullsized":"2024-05-06T00:29:22.520761+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:01:01.894785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":34622764,"num_objects":70,"num_object_clones":16,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":34,"num_read":5359,"num_read_kb":26849,"num_write":617,"num_write_kb":13067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"141'536","reported_seq":6288,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.126240+0000","last_change":"2024-05-06T00:28:32.628240+0000","last_active":"2024-05-06T00:29:22.126240+0000","last_peered":"2024-05-06T00:29:22.126240+0000","last_clean":"2024-05-06T00:29:22.126240+0000","last_became_active":"2024-05-06T00:28:32.626303+0000","last_became_peered":"2024-05-06T00:28:32.626303+0000","last_unstale":"2024-05-06T00:29:22.126240+0000","last_undegraded":"2024-05-06T00:29:22.126240+0000","last_fullsized":"2024-05-06T00:29:22.126240+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:04:55.274018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":48677697,"num_objects":86,"num_object_clones":20,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":41,"num_read":7038,"num_read_kb":32598,"num_write":466,"num_write_kb":5869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"141'275","reported_seq":6476,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.123082+0000","last_change":"2024-05-06T00:28:06.813891+0000","last_active":"2024-05-06T00:29:22.123082+0000","last_peered":"2024-05-06T00:29:22.123082+0000","last_clean":"2024-05-06T00:29:22.123082+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:29:22.123082+0000","last_undegraded":"2024-05-06T00:29:22.123082+0000","last_fullsized":"2024-05-06T00:29:22.123082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:10:37.406925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":34421018,"num_objects":55,"num_object_clones":12,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":29,"num_read":4767,"num_read_kb":20752,"num_write":300,"num_write_kb":6605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"141'321","reported_seq":4860,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.512326+0000","last_change":"2024-05-06T00:29:07.736666+0000","last_active":"2024-05-06T00:29:22.512326+0000","last_peered":"2024-05-06T00:29:22.512326+0000","last_clean":"2024-05-06T00:29:22.512326+0000","last_became_active":"2024-05-06T00:29:07.734795+0000","last_became_peered":"2024-05-06T00:29:07.734795+0000","last_unstale":"2024-05-06T00:29:22.512326+0000","last_undegraded":"2024-05-06T00:29:22.512326+0000","last_fullsized":"2024-05-06T00:29:22.512326+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:43:15.263200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":28415062,"num_objects":61,"num_object_clones":11,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":32,"num_read":4401,"num_read_kb":33224,"num_write":302,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"141'761","reported_seq":10234,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.190604+0000","last_change":"2024-05-06T00:29:19.813529+0000","last_active":"2024-05-06T00:29:22.190604+0000","last_peered":"2024-05-06T00:29:22.190604+0000","last_clean":"2024-05-06T00:29:22.190604+0000","last_became_active":"2024-05-06T00:29:19.811713+0000","last_became_peered":"2024-05-06T00:29:19.811713+0000","last_unstale":"2024-05-06T00:29:22.190604+0000","last_undegraded":"2024-05-06T00:29:22.190604+0000","last_fullsized":"2024-05-06T00:29:22.190604+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:34:48.634829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":26845273,"num_objects":77,"num_object_clones":18,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":43,"num_read":11368,"num_read_kb":21152,"num_write":1073,"num_write_kb":3957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"141'707","reported_seq":11507,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.086496+0000","last_change":"2024-05-06T00:29:07.737933+0000","last_active":"2024-05-06T00:29:22.086496+0000","last_peered":"2024-05-06T00:29:22.086496+0000","last_clean":"2024-05-06T00:29:22.086496+0000","last_became_active":"2024-05-06T00:29:07.735487+0000","last_became_peered":"2024-05-06T00:29:07.735487+0000","last_unstale":"2024-05-06T00:29:22.086496+0000","last_undegraded":"2024-05-06T00:29:22.086496+0000","last_fullsized":"2024-05-06T00:29:22.086496+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:22:19.625919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":41599667,"num_objects":80,"num_object_clones":20,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":38,"num_read":14149,"num_read_kb":23316,"num_write":942,"num_write_kb":4294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"141'274","reported_seq":3489,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.188688+0000","last_change":"2024-05-06T00:28:32.630504+0000","last_active":"2024-05-06T00:29:22.188688+0000","last_peered":"2024-05-06T00:29:22.188688+0000","last_clean":"2024-05-06T00:29:22.188688+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:29:22.188688+0000","last_undegraded":"2024-05-06T00:29:22.188688+0000","last_fullsized":"2024-05-06T00:29:22.188688+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:15:18.346446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":14329331,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":25,"num_read":3116,"num_read_kb":15188,"num_write":325,"num_write_kb":2385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"141'449","reported_seq":10586,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.084401+0000","last_change":"2024-05-06T00:29:07.736408+0000","last_active":"2024-05-06T00:29:22.084401+0000","last_peered":"2024-05-06T00:29:22.084401+0000","last_clean":"2024-05-06T00:29:22.084401+0000","last_became_active":"2024-05-06T00:29:07.734992+0000","last_became_peered":"2024-05-06T00:29:07.734992+0000","last_unstale":"2024-05-06T00:29:22.084401+0000","last_undegraded":"2024-05-06T00:29:22.084401+0000","last_fullsized":"2024-05-06T00:29:22.084401+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:58:46.841900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":38896214,"num_objects":62,"num_object_clones":11,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":30,"num_read":11483,"num_read_kb":26680,"num_write":306,"num_write_kb":4063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"141'352","reported_seq":5204,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.086740+0000","last_change":"2024-05-06T00:28:31.037532+0000","last_active":"2024-05-06T00:29:22.086740+0000","last_peered":"2024-05-06T00:29:22.086740+0000","last_clean":"2024-05-06T00:29:22.086740+0000","last_became_active":"2024-05-06T00:28:31.036279+0000","last_became_peered":"2024-05-06T00:28:31.036279+0000","last_unstale":"2024-05-06T00:29:22.086740+0000","last_undegraded":"2024-05-06T00:29:22.086740+0000","last_fullsized":"2024-05-06T00:29:22.086740+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:33:19.416988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":34381432,"num_objects":60,"num_object_clones":9,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":35,"num_read":5140,"num_read_kb":10401,"num_write":476,"num_write_kb":3507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"141'358","reported_seq":8478,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.031373+0000","last_change":"2024-05-06T00:28:31.628743+0000","last_active":"2024-05-06T00:29:22.031373+0000","last_peered":"2024-05-06T00:29:22.031373+0000","last_clean":"2024-05-06T00:29:22.031373+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:29:22.031373+0000","last_undegraded":"2024-05-06T00:29:22.031373+0000","last_fullsized":"2024-05-06T00:29:22.031373+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:16:26.649861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":50705920,"num_objects":77,"num_object_clones":18,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":43,"num_read":8414,"num_read_kb":21046,"num_write":318,"num_write_kb":3682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"141'431","reported_seq":8548,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-05-06T00:29:22.046807+0000","last_change":"2024-05-06T00:28:31.034254+0000","last_active":"2024-05-06T00:29:22.046807+0000","last_peered":"2024-05-06T00:29:22.046807+0000","last_clean":"2024-05-06T00:29:22.046807+0000","last_became_active":"2024-05-06T00:28:31.032323+0000","last_became_peered":"2024-05-06T00:28:31.032323+0000","last_unstale":"2024-05-06T00:29:22.046807+0000","last_undegraded":"2024-05-06T00:29:22.046807+0000","last_fullsized":"2024-05-06T00:29:22.046807+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:48:16.762201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":40513121,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":23,"num_read":9447,"num_read_kb":38217,"num_write":348,"num_write_kb":5386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1250409765,"num_objects":2047,"num_object_clones":472,"num_object_copies":4094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":1005,"num_read":226637,"num_read_kb":717998,"num_write":15733,"num_write_kb":168783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5546,"num_bytes_recovered":71938434,"num_keys_recovered":904,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118611968,"data_stored":146499146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14557,"ondisk_log_size":14557,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640296,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50308,"kb_used_data":19128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321532,"statfs":{"total":96636764160,"available":96585248768,"internally_reserved":0,"allocated":19587072,"data_stored":20905108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640294,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103888,"kb_used_data":28296,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94267952,"statfs":{"total":96636764160,"available":96530382848,"internally_reserved":0,"allocated":28975104,"data_stored":36390906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640294,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74144,"kb_used_data":20756,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94297696,"statfs":{"total":96636764160,"available":96560840704,"internally_reserved":0,"allocated":21254144,"data_stored":24688659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75036,"kb_used_data":17488,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94296804,"statfs":{"total":96636764160,"available":96559927296,"internally_reserved":0,"allocated":17907712,"data_stored":21432801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71704,"kb_used_data":18316,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94300136,"statfs":{"total":96636764160,"available":96563339264,"internally_reserved":0,"allocated":18755584,"data_stored":23936494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640294,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55892,"kb_used_data":20556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315948,"statfs":{"total":96636764160,"available":96579530752,"internally_reserved":0,"allocated":21049344,"data_stored":22161187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19820544,"data_stored":21904070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17072128,"data_stored":23237415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":21023002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19550208,"data_stored":23982892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":27291648,"data_stored":35691827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":18374656,"data_stored":20659940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:29:24.682 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [2, 3, 0, 1] 2024-05-06T00:29:24.683 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.1', '2', '3', '0', '1'] 2024-05-06T00:29:24.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.1 2 3 0 1 2024-05-06T00:29:24.772 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:24.773+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:24.794 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:24.794 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:24.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:24.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:24.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.794 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:24.794 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 145.99031223599999, 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:29:24.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 145.99029603299999, 2024-05-06T00:29:24.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:29:24.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.982065816000002, 2024-05-06T00:29:24.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.811 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.982034446, 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:29:24.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:29:24.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:29:24.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:29:24.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.982000657, 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:29:24.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981820201000005, 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:29:24.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981786557999996, 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.831 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981762775999997, 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:24.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981736498999993, 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:29:24.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981693258000007, 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:29:24.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:29:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:29:24.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:29:24.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981663628999996, 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:24.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.856 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981630779, 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:29:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:29:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981599141000004, 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:29:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:24.867 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981566338999997, 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:24.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:29:24.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981534306, 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.873 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:29:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981502226999993, 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.879 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:29:24.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:29:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.981470919000003, 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:29:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.888 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.141325202000001, 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:24.889 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:29:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:29:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.139919053, 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:29:24.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:29:24.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.139866094999999, 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.903 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:24.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:24.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:24.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:29:24.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:29:24.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:24.908 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:24.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:24.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:29:24.910 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:24.910+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:24.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:25.002 INFO:teuthology.orchestra.run.smithi186.stdout:truncating image image_client.0-clone40 from 0xec41a00 (overlap 0x735400) to 0x39aa00 2024-05-06T00:29:25.040 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:25.041+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:25.112 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:25.113+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:25.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:25.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:25.132 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:25.133 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:25.133 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:25.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:25.150 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:25.150 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:25.173 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:25.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:25.213 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:25.214+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:25.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:25.218 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:25.267 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:25.267 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:25.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:25.313 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:25.314+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:25.414 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:25.414+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:25.450 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:25.451 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:25.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:25.515 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:25.513+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:25.593 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:25.593 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:25.607 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:25.645 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #39, image image_client.0-clone39 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent40 2024-05-06T00:29:25.648 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:25.649+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:25.717 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:25.718+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:25.785 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:25.785 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:25.799 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:25.818 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:25.818+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:25.835 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:29:25.838 INFO:teuthology.orchestra.run.smithi078.stderr:set 2.1 pg_upmap_items mapping to [2->3,0->1] 2024-05-06T00:29:25.919 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:25.918+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:25.931 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:25.931 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:25.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:26.020 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:26.021+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:26.101 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:26.101 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:26.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:26.121 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:26.121+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:26.222 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:26.222+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:26.241 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:26.241 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:26.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:26.272 INFO:teuthology.orchestra.run.smithi186.stdout:5425 write 0xe9c0a00 thru 0xe9cb66c (0xac6d bytes) 2024-05-06T00:29:26.274 INFO:teuthology.orchestra.run.smithi186.stdout:5427 punch from 0x206f400 to 0x2077000, (0x7c00 bytes) 2024-05-06T00:29:26.345 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:26.346+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:26.379 INFO:teuthology.orchestra.run.smithi186.stdout:5431 trunc from 0xec41a00 to 0x3acce00 2024-05-06T00:29:26.418 INFO:teuthology.orchestra.run.smithi186.stdout:5432 read 0x70ec00 thru 0x7144b7 (0x58b8 bytes) 2024-05-06T00:29:26.421 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:26.421 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:26.424 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:26.422+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:26.426 INFO:teuthology.orchestra.run.smithi186.stdout:5433 read 0x10bb600 thru 0x10c282f (0x7230 bytes) 2024-05-06T00:29:26.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:26.452 INFO:teuthology.orchestra.run.smithi186.stdout:5437 write 0x4e94000 thru 0x4e98386 (0x4387 bytes) 2024-05-06T00:29:26.454 INFO:teuthology.orchestra.run.smithi186.stdout:5438 write 0x6000 thru 0xf819 (0x981a bytes) 2024-05-06T00:29:26.455 INFO:teuthology.orchestra.run.smithi186.stdout:5439 read 0x6ac200 thru 0x6b00fd (0x3efe bytes) 2024-05-06T00:29:26.459 INFO:teuthology.orchestra.run.smithi186.stdout:5441 write 0x31b2600 thru 0x31bbcbc (0x96bd bytes) 2024-05-06T00:29:26.459 INFO:teuthology.orchestra.run.smithi186.stdout:5445 read 0x1d9ae00 thru 0x1da3cb4 (0x8eb5 bytes) 2024-05-06T00:29:26.460 INFO:teuthology.orchestra.run.smithi186.stdout:5447 trunc from 0x4e98400 to 0x53a7600 2024-05-06T00:29:26.524 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:26.525+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:26.588 INFO:teuthology.orchestra.run.smithi186.stdout:5450 write 0x948ce00 thru 0x94917c7 (0x49c8 bytes) 2024-05-06T00:29:26.590 INFO:teuthology.orchestra.run.smithi186.stdout:5451 write 0x5639e00 thru 0x56427b9 (0x89ba bytes) 2024-05-06T00:29:26.590 INFO:teuthology.orchestra.run.smithi186.stdout:5452 punch from 0x3993400 to 0x399ec00, (0xb800 bytes) 2024-05-06T00:29:26.592 INFO:teuthology.orchestra.run.smithi186.stdout:5453 punch from 0x44a1c00 to 0x44ae400, (0xc800 bytes) 2024-05-06T00:29:26.592 INFO:teuthology.orchestra.run.smithi186.stdout:5455 write 0xd25000 thru 0xd2a824 (0x5825 bytes) 2024-05-06T00:29:26.593 INFO:teuthology.orchestra.run.smithi186.stdout:5457 read 0x8f1f200 thru 0x8f1f300 (0x101 bytes) 2024-05-06T00:29:26.612 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:26.612 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:26.625 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:26.626+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:26.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:26.641 INFO:teuthology.orchestra.run.smithi186.stdout:5461 write 0x956a200 thru 0x9576815 (0xc616 bytes) 2024-05-06T00:29:26.644 INFO:teuthology.orchestra.run.smithi186.stdout:5464 read 0x20da000 thru 0x20dd530 (0x3531 bytes) 2024-05-06T00:29:26.645 INFO:teuthology.orchestra.run.smithi186.stdout:5472 punch from 0x1a9a400 to 0x1a9c200, (0x1e00 bytes) 2024-05-06T00:29:26.647 INFO:teuthology.orchestra.run.smithi186.stdout:5473 read 0x73d9c00 thru 0x73da694 (0xa95 bytes) 2024-05-06T00:29:26.647 INFO:teuthology.orchestra.run.smithi186.stdout:5474 write 0x358d000 thru 0x35981c8 (0xb1c9 bytes) 2024-05-06T00:29:26.647 INFO:teuthology.orchestra.run.smithi186.stdout:5476 read 0x3637200 thru 0x3642d0e (0xbb0f bytes) 2024-05-06T00:29:26.651 INFO:teuthology.orchestra.run.smithi186.stdout:5477 trunc from 0x9576a00 to 0x7e86600 2024-05-06T00:29:26.721 INFO:teuthology.orchestra.run.smithi186.stdout:5478 write 0xa315400 thru 0xa3185c1 (0x31c2 bytes) 2024-05-06T00:29:26.726 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:26.726+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:26.727 INFO:teuthology.orchestra.run.smithi186.stdout:5481 read 0x222b000 thru 0x222e145 (0x3146 bytes) 2024-05-06T00:29:26.742 INFO:teuthology.orchestra.run.smithi186.stdout:5482 trunc from 0xa318600 to 0x4e8ba00 2024-05-06T00:29:26.788 INFO:teuthology.orchestra.run.smithi186.stdout:5485 punch from 0x38b7200 to 0x38bba00, (0x4800 bytes) 2024-05-06T00:29:26.791 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:26.791 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:26.802 INFO:teuthology.orchestra.run.smithi186.stdout:5486 trunc from 0x4e8ba00 to 0xa996000 2024-05-06T00:29:26.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:26.849 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:26.850+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:26.859 INFO:teuthology.orchestra.run.smithi186.stdout:5487 write 0xab54c00 thru 0xab55bdd (0xfde bytes) 2024-05-06T00:29:26.864 INFO:teuthology.orchestra.run.smithi186.stdout:5488 read 0x4d19800 thru 0x4d20b29 (0x732a bytes) 2024-05-06T00:29:26.865 INFO:teuthology.orchestra.run.smithi186.stdout:5489 write 0x38a4a00 thru 0x38a7d52 (0x3353 bytes) 2024-05-06T00:29:26.865 INFO:teuthology.orchestra.run.smithi186.stdout:5490 read 0x43ed800 thru 0x43f746c (0x9c6d bytes) 2024-05-06T00:29:26.867 INFO:teuthology.orchestra.run.smithi186.stdout:5491 punch from 0x90d1400 to 0x90d6600, (0x5200 bytes) 2024-05-06T00:29:26.868 INFO:teuthology.orchestra.run.smithi186.stdout:5495 punch from 0xa1a4600 to 0xa1b2000, (0xda00 bytes) 2024-05-06T00:29:26.868 INFO:teuthology.orchestra.run.smithi186.stdout:5497 write 0x4487400 thru 0x448b2e4 (0x3ee5 bytes) 2024-05-06T00:29:26.900 INFO:teuthology.orchestra.run.smithi186.stdout:5499 write 0xb2f5000 thru 0xb2f7796 (0x2797 bytes) 2024-05-06T00:29:26.902 INFO:teuthology.orchestra.run.smithi186.stdout:5501 read 0x4ad2a00 thru 0x4ad9ab8 (0x70b9 bytes) 2024-05-06T00:29:26.902 INFO:teuthology.orchestra.run.smithi186.stdout:5503 read 0x578a200 thru 0x578bd8c (0x1b8d bytes) 2024-05-06T00:29:26.902 INFO:teuthology.orchestra.run.smithi186.stdout:5507 read 0x11800 thru 0x2135c (0xfb5d bytes) 2024-05-06T00:29:26.905 INFO:teuthology.orchestra.run.smithi186.stdout:5511 read 0x9b8aa00 thru 0x9b8f3ad (0x49ae bytes) 2024-05-06T00:29:26.905 INFO:teuthology.orchestra.run.smithi186.stdout:5513 write 0x1b36000 thru 0x1b36134 (0x135 bytes) 2024-05-06T00:29:26.905 INFO:teuthology.orchestra.run.smithi186.stdout:5515 read 0x9818600 thru 0x9824523 (0xbf24 bytes) 2024-05-06T00:29:26.905 INFO:teuthology.orchestra.run.smithi186.stdout:5516 write 0x8d600 thru 0x9bafc (0xe4fd bytes) 2024-05-06T00:29:26.908 INFO:teuthology.orchestra.run.smithi186.stdout:5517 write 0x89c4400 thru 0x89c60fe (0x1cff bytes) 2024-05-06T00:29:26.908 INFO:teuthology.orchestra.run.smithi186.stdout:5518 read 0x6659600 thru 0x665e7e2 (0x51e3 bytes) 2024-05-06T00:29:26.909 INFO:teuthology.orchestra.run.smithi186.stdout:5522 write 0x48f5000 thru 0x48fb5ac (0x65ad bytes) 2024-05-06T00:29:26.909 INFO:teuthology.orchestra.run.smithi186.stdout:5524 write 0x95fa000 thru 0x9601298 (0x7299 bytes) 2024-05-06T00:29:26.909 INFO:teuthology.orchestra.run.smithi186.stdout:5525 read 0xae06800 thru 0xae0cca8 (0x64a9 bytes) 2024-05-06T00:29:26.909 INFO:teuthology.orchestra.run.smithi186.stdout:5526 read 0x240fc00 thru 0x24124dd (0x28de bytes) 2024-05-06T00:29:26.909 INFO:teuthology.orchestra.run.smithi186.stdout:5527 read 0x358c000 thru 0x35974b8 (0xb4b9 bytes) 2024-05-06T00:29:26.910 INFO:teuthology.orchestra.run.smithi186.stdout:5529 read 0x1246800 thru 0x1251c62 (0xb463 bytes) 2024-05-06T00:29:26.928 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:26.926+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:26.973 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:26.973 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:26.977 INFO:teuthology.orchestra.run.smithi186.stdout:5531 trunc from 0xb2f7800 to 0x1b6600 2024-05-06T00:29:26.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:27.029 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:27.029+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:27.130 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:27.129+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:27.158 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:27.158 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:27.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:27.194 INFO:teuthology.orchestra.run.smithi186.stdout:5533 write 0x1052800 thru 0x1055554 (0x2d55 bytes) 2024-05-06T00:29:27.196 INFO:teuthology.orchestra.run.smithi186.stdout:5535 write 0x898800 thru 0x89ea2f (0x6230 bytes) 2024-05-06T00:29:27.231 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:27.229+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:27.236 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:27.237 INFO:teuthology.orchestra.run.smithi186.stdout:5537 write 0x643bc00 thru 0x6448a67 (0xce68 bytes) 2024-05-06T00:29:27.282 INFO:teuthology.orchestra.run.smithi186.stdout:5544 write 0xe785e00 thru 0xe78c5bd (0x67be bytes) 2024-05-06T00:29:27.283 INFO:teuthology.orchestra.run.smithi186.stdout:5545 write 0xe4c3000 thru 0xe4d00cb (0xd0cc bytes) 2024-05-06T00:29:27.299 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:27.299 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:27.312 INFO:teuthology.orchestra.run.smithi186.stdout:5548 trunc from 0xe78c600 to 0x3189000 2024-05-06T00:29:27.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:27.357 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:27.358+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:27.372 INFO:teuthology.orchestra.run.smithi186.stdout:5549 write 0x3769800 thru 0x376dcb5 (0x44b6 bytes) 2024-05-06T00:29:27.375 INFO:teuthology.orchestra.run.smithi186.stdout:5550 read 0x1d6a400 thru 0x1d76f42 (0xcb43 bytes) 2024-05-06T00:29:27.397 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:27.398 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:27.412 INFO:teuthology.orchestra.run.smithi186.stdout:5551 write 0xaab8000 thru 0xaabcb01 (0x4b02 bytes) 2024-05-06T00:29:27.413 INFO:teuthology.orchestra.run.smithi186.stdout:5552 write 0x1791c00 thru 0x179683e (0x4c3f bytes) 2024-05-06T00:29:27.413 INFO:teuthology.orchestra.run.smithi186.stdout:5553 punch from 0x81c6400 to 0x81d1e00, (0xba00 bytes) 2024-05-06T00:29:27.414 INFO:teuthology.orchestra.run.smithi186.stdout:5554 punch from 0x21a2400 to 0x21a6400, (0x4000 bytes) 2024-05-06T00:29:27.415 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:27.415 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:27.424 INFO:teuthology.orchestra.run.smithi186.stdout:5560 trunc from 0xaabcc00 to 0xebe6e00 2024-05-06T00:29:27.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:27.433 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:27.434+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:27.442 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:27.442 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:27.454 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:27.459 INFO:teuthology.orchestra.run.smithi186.stdout:5561 trunc from 0xebe6e00 to 0x8080a00 2024-05-06T00:29:27.460 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:27.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:27.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:27.501 INFO:teuthology.orchestra.run.smithi186.stdout:5567 write 0xada2e00 thru 0xada75f2 (0x47f3 bytes) 2024-05-06T00:29:27.502 INFO:teuthology.orchestra.run.smithi186.stdout:5569 read 0x62cca00 thru 0x62ced70 (0x2371 bytes) 2024-05-06T00:29:27.502 INFO:teuthology.orchestra.run.smithi186.stdout:5572 punch from 0x740f000 to 0x740f600, (0x600 bytes) 2024-05-06T00:29:27.503 INFO:teuthology.orchestra.run.smithi186.stdout:5573 write 0x88b1600 thru 0x88b7c4e (0x664f bytes) 2024-05-06T00:29:27.503 INFO:teuthology.orchestra.run.smithi186.stdout:5577 read 0x20a800 thru 0x20f5b1 (0x4db2 bytes) 2024-05-06T00:29:27.503 INFO:teuthology.orchestra.run.smithi186.stdout:5578 read 0x785dc00 thru 0x7868ad8 (0xaed9 bytes) 2024-05-06T00:29:27.513 INFO:teuthology.orchestra.run.smithi186.stdout:5579 trunc from 0xada7600 to 0x5061200 2024-05-06T00:29:27.539 INFO:teuthology.orchestra.run.smithi186.stdout:5580 punch from 0xa1fe00 to 0xa29e00, (0xa000 bytes) 2024-05-06T00:29:27.543 INFO:teuthology.orchestra.run.smithi186.stdout:5583 trunc from 0x5061200 to 0x6e64000 2024-05-06T00:29:27.560 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:27.561+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:27.569 INFO:teuthology.orchestra.run.smithi186.stdout:5585 trunc from 0x6e64000 to 0x2dcd600 2024-05-06T00:29:27.582 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:27.582 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:27.597 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:27.624 INFO:teuthology.orchestra.run.smithi186.stdout:5587 write 0x325f800 thru 0x3269919 (0xa11a bytes) 2024-05-06T00:29:27.640 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:27.641+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:27.647 INFO:teuthology.orchestra.run.smithi186.stdout:5588 trunc from 0x3269a00 to 0xc3a4a00 2024-05-06T00:29:27.675 INFO:teuthology.orchestra.run.smithi186.stdout:5589 punch from 0x3f8e400 to 0x3f97800, (0x9400 bytes) 2024-05-06T00:29:27.678 INFO:teuthology.orchestra.run.smithi186.stdout:5592 write 0x8c52400 thru 0x8c5e270 (0xbe71 bytes) 2024-05-06T00:29:27.678 INFO:teuthology.orchestra.run.smithi186.stdout:5593 clone 42 order 20 su 32768 sc 4 2024-05-06T00:29:27.735 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:27.734+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:27.766 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:27.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:27.767 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:27.767 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:27.767 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:27.780 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:27.836 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:27.834+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:27.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:29:27.910 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:27.910 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:27.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:27.911 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:27.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.911 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:27.911 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:27.911 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 149.10644466100001, 2024-05-06T00:29:27.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:29:27.911 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:29:27.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 149.10642845800001, 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:29:27.917 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:29:27.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:29:27.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:27.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.098198241000006, 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.924 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:29:27.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.925 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:27.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:27.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:29:27.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:29:27.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.098166871000004, 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.931 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:29:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.098133082000004, 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.936 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:29:27.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:29:27.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:29:27.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097952625999994, 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097918983, 2024-05-06T00:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:29:27.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:29:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097895201, 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097868923999997, 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097825682999996, 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:29:27.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.966 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097796054, 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:29:27.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097763204000003, 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.972 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097731565999993, 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:29:27.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:29:27.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097698764, 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:29:27.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:29:27.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097666731000004, 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:29:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097634651999996, 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:29:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.097603344000007, 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:29:27.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.257457627000001, 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.002 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:29:28.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:29:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.256051478000003, 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:29:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:29:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.255998519999999, 2024-05-06T00:29:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:29:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:29:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.020 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:28.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:28.022 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:28.021+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:28.052 INFO:teuthology.orchestra.run.smithi186.stdout:leaving image image_client.0-clone41 intact 2024-05-06T00:29:28.123 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:28.122+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:28.178 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:28.178 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:28.178 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:28.178 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:28.191 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:28.236 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:28.237+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:28.310 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:28.310 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:28.310 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:28.310 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:28.310 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:28.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:28.325 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:28.325+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:28.425 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:28.426+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:28.500 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472291686999995, 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.501 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472270898000005, 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:28.510 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472250836000001, 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:29:28.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472231559999997, 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.515 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:28.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472211442000003, 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:29:28.519 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:29:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.523 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472180033000001, 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.524 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:28.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:29:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472147121000006, 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:29:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472126533999997, 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:29:28.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472107154, 2024-05-06T00:29:28.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:29:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.472062472999994, 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.543 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:29:28.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.471997381999998, 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:29:28.547 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:29:28.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.471965807999993, 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:29:28.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:29:28.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:28.556 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.471943546999995, 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:28.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.471922430999996, 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:29:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:29:28.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.814205805, 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:29:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:29:28.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.813795562999999, 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.875150969000003, 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:29:28.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:29:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:29:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.875079685000003, 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:28.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.584 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.875048051999997, 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:29:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:29:28.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:28.590 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.870829506, 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:29:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:29:28.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:29:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:28.597 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:28.599 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:28.599 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:28.598+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:28.700 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:28.701+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:28.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:28.729 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:28.729 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:28.729 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:28.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:28.832 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:28.833+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:28.848 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #40, image image_client.0-clone40 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent41 2024-05-06T00:29:28.859 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:29:28.923 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:28.923 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:28.923 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:28.924 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:28.924 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:28.924 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:28.921+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:28.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:28.989 INFO:teuthology.orchestra.run.smithi186.stdout:5595 write 0x8556c00 thru 0x8561766 (0xab67 bytes) 2024-05-06T00:29:28.990 INFO:teuthology.orchestra.run.smithi186.stdout:5596 write 0x13d1600 thru 0x13da543 (0x8f44 bytes) 2024-05-06T00:29:28.992 INFO:teuthology.orchestra.run.smithi186.stdout:5599 write 0xc62a00 thru 0xc69e43 (0x7444 bytes) 2024-05-06T00:29:28.998 INFO:teuthology.orchestra.run.smithi186.stdout:5600 punch from 0x8763c00 to 0x8769c00, (0x6000 bytes) 2024-05-06T00:29:29.028 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:29.029+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:29.035 INFO:teuthology.orchestra.run.smithi186.stdout:5602 write 0x263da00 thru 0x2642140 (0x4741 bytes) 2024-05-06T00:29:29.036 INFO:teuthology.orchestra.run.smithi186.stdout:5603 read 0x31d8a00 thru 0x31dd2f9 (0x48fa bytes) 2024-05-06T00:29:29.036 INFO:teuthology.orchestra.run.smithi186.stdout:5606 punch from 0x2a91a00 to 0x2a9e000, (0xc600 bytes) 2024-05-06T00:29:29.052 INFO:teuthology.orchestra.run.smithi186.stdout:5609 read 0x1f35200 thru 0x1f3bedd (0x6cde bytes) 2024-05-06T00:29:29.053 INFO:teuthology.orchestra.run.smithi186.stdout:5610 punch from 0x8dcba00 to 0x8dcea00, (0x3000 bytes) 2024-05-06T00:29:29.067 INFO:teuthology.orchestra.run.smithi186.stdout:5612 trunc from 0xc3a4a00 to 0xcb01600 2024-05-06T00:29:29.091 INFO:teuthology.orchestra.run.smithi186.stdout:5614 punch from 0x88f4e00 to 0x8904000, (0xf200 bytes) 2024-05-06T00:29:29.103 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:29.101+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:29.109 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.10310035099999, 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:29:29.110 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:29:29.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:29:29.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.102871779, 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:29:29.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:29:29.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.102794495, 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:29:29.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.10259851399999, 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:29:29.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:29:29.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.195489137999999, 2024-05-06T00:29:29.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:29:29.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.195428258999996, 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.133 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:29:29.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:29:29.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:29:29.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.077871809000001, 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.137 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:29.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:29.141 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.077827080999995, 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:29:29.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.077788679999998, 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.077751418000005, 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:29:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:29:29.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.077714509000003, 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:29:29.155 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:29:29.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.077676590999999, 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:29:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.077639476000002, 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:29:29.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:29:29.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.071859027000002, 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:29:29.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.071820943000006, 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.173 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:29.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:29:29.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:29.177 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.071783268999994, 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:29:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.062347759000005, 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:29:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:29:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:29:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:29:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.062315331999997, 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:29:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:29:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:29:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:29:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.062291450000004, 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:29:29.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:29.198 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.062271519999996, 2024-05-06T00:29:29.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:29:29.199 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:29.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:29.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:29.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:29.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:29.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:29.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:29:29.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.200 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:29.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:29.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:29.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.204 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:29.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:29.205 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:29.206 INFO:teuthology.orchestra.run.smithi186.stdout:5616 trunc from 0xcb01600 to 0x9104800 2024-05-06T00:29:29.206 INFO:teuthology.orchestra.run.smithi186.stdout:5619 read 0x66a600 thru 0x670ec5 (0x68c6 bytes) 2024-05-06T00:29:29.206 INFO:teuthology.orchestra.run.smithi186.stdout:5620 trunc from 0x9104800 to 0x8eaec00 2024-05-06T00:29:29.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:29.207 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:29.206+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:29.276 INFO:teuthology.orchestra.run.smithi186.stdout:5621 trunc from 0x8eaec00 to 0xedf9800 2024-05-06T00:29:29.296 INFO:teuthology.orchestra.run.smithi186.stdout:5622 read 0xc97dc00 thru 0xc985578 (0x7979 bytes) 2024-05-06T00:29:29.298 INFO:teuthology.orchestra.run.smithi186.stdout:5623 read 0xcb8fa00 thru 0xcb9770f (0x7d10 bytes) 2024-05-06T00:29:29.298 INFO:teuthology.orchestra.run.smithi186.stdout:5625 write 0x89f6400 thru 0x89f8f59 (0x2b5a bytes) 2024-05-06T00:29:29.299 INFO:teuthology.orchestra.run.smithi186.stdout:5626 punch from 0xaec7200 to 0xaece200, (0x7000 bytes) 2024-05-06T00:29:29.300 INFO:teuthology.orchestra.run.smithi186.stdout:5631 punch from 0xdd10400 to 0xdd1a000, (0x9c00 bytes) 2024-05-06T00:29:29.300 INFO:teuthology.orchestra.run.smithi186.stdout:5633 read 0x1e29a00 thru 0x1e33b35 (0xa136 bytes) 2024-05-06T00:29:29.300 INFO:teuthology.orchestra.run.smithi186.stdout:5634 punch from 0xe70d800 to 0xe71a000, (0xc800 bytes) 2024-05-06T00:29:29.306 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:29.306+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:29.347 INFO:teuthology.orchestra.run.smithi186.stdout:5635 trunc from 0xedf9800 to 0x2e02c00 2024-05-06T00:29:29.349 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:29.349 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:29.349 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:29.349 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:29.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:29.395 INFO:teuthology.orchestra.run.smithi186.stdout:5637 write 0x1fea200 thru 0x1ff1548 (0x7349 bytes) 2024-05-06T00:29:29.397 INFO:teuthology.orchestra.run.smithi186.stdout:5638 punch from 0x1102e00 to 0x1109000, (0x6200 bytes) 2024-05-06T00:29:29.408 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:29.409+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:29.422 INFO:teuthology.orchestra.run.smithi186.stdout:5639 trunc from 0x2e02c00 to 0xebcaa00 2024-05-06T00:29:29.438 INFO:teuthology.orchestra.run.smithi186.stdout:5640 read 0x8b80200 thru 0x8b8eb63 (0xe964 bytes) 2024-05-06T00:29:29.469 INFO:teuthology.orchestra.run.smithi186.stdout:5641 trunc from 0xebcaa00 to 0x1a88c00 2024-05-06T00:29:29.509 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:29.509+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:29.514 INFO:teuthology.orchestra.run.smithi186.stdout:5646 punch from 0xebba00 to 0xec2c00, (0x7200 bytes) 2024-05-06T00:29:29.516 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:29.517 INFO:teuthology.orchestra.run.smithi186.stdout:5648 trunc from 0x1a88c00 to 0x214f400 2024-05-06T00:29:29.550 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:29.550 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:29.550 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:29.551 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:29.551 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:29.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:29.569 INFO:teuthology.orchestra.run.smithi186.stdout:5649 write 0x7e17800 thru 0x7e1ca95 (0x5296 bytes) 2024-05-06T00:29:29.571 INFO:teuthology.orchestra.run.smithi186.stdout:5650 read 0x3ec5400 thru 0x3ecef4f (0x9b50 bytes) 2024-05-06T00:29:29.572 INFO:teuthology.orchestra.run.smithi186.stdout:5651 write 0xfe7000 thru 0xff290b (0xb90c bytes) 2024-05-06T00:29:29.592 INFO:teuthology.orchestra.run.smithi186.stdout:5652 trunc from 0x7e1cc00 to 0x1664a00 2024-05-06T00:29:29.610 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:29.609+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:29.644 INFO:teuthology.orchestra.run.smithi186.stdout:5653 write 0x954e00 thru 0x960c35 (0xbe36 bytes) 2024-05-06T00:29:29.647 INFO:teuthology.orchestra.run.smithi186.stdout:5654 read 0x206e00 thru 0x2129b0 (0xbbb1 bytes) 2024-05-06T00:29:29.648 INFO:teuthology.orchestra.run.smithi186.stdout:5656 punch from 0x101b600 to 0x1026000, (0xaa00 bytes) 2024-05-06T00:29:29.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:29.678 INFO:teuthology.orchestra.run.smithi186.stdout:5657 write 0x2029000 thru 0x203559c (0xc59d bytes) 2024-05-06T00:29:29.679 INFO:teuthology.orchestra.run.smithi186.stdout:5658 read 0x6df600 thru 0x6ea654 (0xb055 bytes) 2024-05-06T00:29:29.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:29.706 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:29.706 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:29.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:29.706 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:29.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.706 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:29:29.706 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 180.10315021400001, 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:29:29.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:29:29.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.830466339, 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:29:29.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.83037766000001, 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:29:29.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:29:29.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.83027571700001, 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.726 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:29:29.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:29:29.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.830166615, 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:29:29.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:29:29.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.830068756, 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:29.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:29:29.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.745 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.83000370600001, 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:29:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:29:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.82991106, 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:29:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:29:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.829839056, 2024-05-06T00:29:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:29:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.07037757000001, 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:29:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.070358322, 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:29:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:29:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.07030820200001, 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:29:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:29:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.070287151, 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:29:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.070271832, 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:29:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.07025494999999, 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:29:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.07023661400001, 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:29:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.07022061200001, 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.07013424900001, 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [read 12582912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.890884003, 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857773900000002, 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.812763+0000", 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99789039000000002 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813015+0000", 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025184099999999999 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813243+0000", 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022833400000000001 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [sparse-read 12587008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.890783531, 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99922042799999999, 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813844+0000", 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99888496199999999 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813901+0000", 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6561000000000003e-05 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813986+0000", 2024-05-06T00:29:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5643999999999995e-05 2024-05-06T00:29:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:29.826 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:29.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:29.828 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:29.829 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:29.829 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:29.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:29.830 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:29.826+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:29.830 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:29.831 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:29.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:29.832 INFO:teuthology.orchestra.run.smithi186.stdout:5659 write 0x8b24600 thru 0x8b29111 (0x4b12 bytes) 2024-05-06T00:29:29.832 INFO:teuthology.orchestra.run.smithi186.stdout:5665 read 0x5869600 thru 0x58782a1 (0xeca2 bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5670 punch from 0x55f0600 to 0x55fda00, (0xd400 bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5672 read 0x4c32600 thru 0x4c3d73d (0xb13e bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5674 read 0x7510400 thru 0x75147e2 (0x43e3 bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5675 read 0x5f87e00 thru 0x5f89f5b (0x215c bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5677 punch from 0x82ff600 to 0x830a600, (0xb000 bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5678 read 0x19a0200 thru 0x19a253c (0x233d bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5682 punch from 0x36bc00 to 0x379a00, (0xde00 bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5683 read 0x757bc00 thru 0x75896be (0xdabf bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5685 punch from 0x8a54600 to 0x8a5d400, (0x8e00 bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5689 write 0xdf43e00 thru 0xdf4cc3d (0x8e3e bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5691 punch from 0x7498600 to 0x74a7200, (0xec00 bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5693 punch from 0x9c66e00 to 0x9c73a00, (0xcc00 bytes) 2024-05-06T00:29:29.833 INFO:teuthology.orchestra.run.smithi186.stdout:5694 write 0x426d800 thru 0x426e9c7 (0x11c8 bytes) 2024-05-06T00:29:29.834 INFO:teuthology.orchestra.run.smithi186.stdout:5696 trunc from 0xdf4ce00 to 0x956b600 2024-05-06T00:29:29.834 INFO:teuthology.orchestra.run.smithi186.stdout:5698 write 0xcff6200 thru 0xcffe9ba (0x87bb bytes) 2024-05-06T00:29:29.834 INFO:teuthology.orchestra.run.smithi186.stdout:5699 write 0xb728a00 thru 0xb730d6b (0x836c bytes) 2024-05-06T00:29:29.834 INFO:teuthology.orchestra.run.smithi186.stdout:5701 write 0x3aac000 thru 0x3aac2a9 (0x2aa bytes) 2024-05-06T00:29:29.834 INFO:teuthology.orchestra.run.smithi186.stdout:5704 trunc from 0xcffea00 to 0xbd03800 2024-05-06T00:29:29.834 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:29.846 INFO:teuthology.orchestra.run.smithi186.stdout:5705 read 0x2e79e00 thru 0x2e7f4e9 (0x56ea bytes) 2024-05-06T00:29:29.848 INFO:teuthology.orchestra.run.smithi186.stdout:5707 punch from 0x6401000 to 0x6409800, (0x8800 bytes) 2024-05-06T00:29:29.848 INFO:teuthology.orchestra.run.smithi186.stdout:5708 read 0x614ce00 thru 0x615bf77 (0xf178 bytes) 2024-05-06T00:29:29.848 INFO:teuthology.orchestra.run.smithi186.stdout:5709 read 0xcab000 thru 0xcb2216 (0x7217 bytes) 2024-05-06T00:29:29.849 INFO:teuthology.orchestra.run.smithi186.stdout:5710 write 0x68f3c00 thru 0x68f6de8 (0x31e9 bytes) 2024-05-06T00:29:29.849 INFO:teuthology.orchestra.run.smithi186.stdout:5713 read 0x3e4e000 thru 0x3e5c39f (0xe3a0 bytes) 2024-05-06T00:29:29.850 INFO:teuthology.orchestra.run.smithi186.stdout:5714 read 0x561fc00 thru 0x562a223 (0xa624 bytes) 2024-05-06T00:29:29.869 INFO:teuthology.orchestra.run.smithi186.stdout:5716 trunc from 0xbd03800 to 0x4d4a600 2024-05-06T00:29:29.906 INFO:teuthology.orchestra.run.smithi186.stdout:5718 read 0x3941800 thru 0x394989e (0x809f bytes) 2024-05-06T00:29:29.908 INFO:teuthology.orchestra.run.smithi186.stdout:5719 punch from 0x22dd400 to 0x22df800, (0x2400 bytes) 2024-05-06T00:29:29.909 INFO:teuthology.orchestra.run.smithi186.stdout:5720 write 0x126ec00 thru 0x1279b04 (0xaf05 bytes) 2024-05-06T00:29:29.910 INFO:teuthology.orchestra.run.smithi186.stdout:5722 read 0x1a20c00 thru 0x1a26a95 (0x5e96 bytes) 2024-05-06T00:29:29.928 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:29.929+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:29.930 INFO:teuthology.orchestra.run.smithi186.stdout:5723 trunc from 0x4d4a600 to 0x1cc200 2024-05-06T00:29:29.981 INFO:teuthology.orchestra.run.smithi186.stdout:5724 punch from 0x7b800 to 0x8ae00, (0xf600 bytes) 2024-05-06T00:29:29.983 INFO:teuthology.orchestra.run.smithi186.stdout:5725 punch from 0x110400 to 0x11ce00, (0xca00 bytes) 2024-05-06T00:29:30.022 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:30.022 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:30.029 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:30.029+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:30.031 INFO:teuthology.orchestra.run.smithi186.stdout:5726 write 0xc48d400 thru 0xc49b015 (0xdc16 bytes) 2024-05-06T00:29:30.032 INFO:teuthology.orchestra.run.smithi186.stdout:5728 read 0x60ec000 thru 0x60ecb20 (0xb21 bytes) 2024-05-06T00:29:30.033 INFO:teuthology.orchestra.run.smithi186.stdout:5731 write 0x1dc9000 thru 0x1dd4972 (0xb973 bytes) 2024-05-06T00:29:30.033 INFO:teuthology.orchestra.run.smithi186.stdout:5734 write 0xa3ec400 thru 0xa3f13e8 (0x4fe9 bytes) 2024-05-06T00:29:30.033 INFO:teuthology.orchestra.run.smithi186.stdout:5735 read 0x2b03400 thru 0x2b067ed (0x33ee bytes) 2024-05-06T00:29:30.033 INFO:teuthology.orchestra.run.smithi186.stdout:5736 read 0x9c16800 thru 0x9c1b9e7 (0x51e8 bytes) 2024-05-06T00:29:30.035 INFO:teuthology.orchestra.run.smithi186.stdout:5738 trunc from 0xc49b200 to 0xd1ec000 2024-05-06T00:29:30.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:30.073 INFO:teuthology.orchestra.run.smithi186.stdout:5741 write 0x3b3fc00 thru 0x3b4c0e3 (0xc4e4 bytes) 2024-05-06T00:29:30.083 INFO:teuthology.orchestra.run.smithi186.stdout:5742 trunc from 0xd1ec000 to 0x926c000 2024-05-06T00:29:30.130 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:30.129+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:30.145 INFO:teuthology.orchestra.run.smithi186.stdout:5743 write 0xad65800 thru 0xad75773 (0xff74 bytes) 2024-05-06T00:29:30.146 INFO:teuthology.orchestra.run.smithi186.stdout:5748 punch from 0x5439400 to 0x5448000, (0xec00 bytes) 2024-05-06T00:29:30.161 INFO:teuthology.orchestra.run.smithi186.stdout:5749 trunc from 0xad75800 to 0x41e5800 2024-05-06T00:29:30.173 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:30.173 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:30.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:30.193 INFO:teuthology.orchestra.run.smithi186.stdout:5750 trunc from 0x41e5800 to 0x2cc5000 2024-05-06T00:29:30.231 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:30.229+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:30.255 INFO:teuthology.orchestra.run.smithi186.stdout:5753 write 0x4b0f200 thru 0x4b1acf3 (0xbaf4 bytes) 2024-05-06T00:29:30.256 INFO:teuthology.orchestra.run.smithi186.stdout:5755 read 0x3fb2000 thru 0x3fbf10e (0xd10f bytes) 2024-05-06T00:29:30.256 INFO:teuthology.orchestra.run.smithi186.stdout:5756 write 0x3a4a00 thru 0x3aa6b3 (0x5cb4 bytes) 2024-05-06T00:29:30.271 INFO:teuthology.orchestra.run.smithi186.stdout:5757 trunc from 0x4b1ae00 to 0x9be4600 2024-05-06T00:29:30.300 INFO:teuthology.orchestra.run.smithi186.stdout:5760 read 0x26c1200 thru 0x26ca512 (0x9313 bytes) 2024-05-06T00:29:30.302 INFO:teuthology.orchestra.run.smithi186.stdout:5761 write 0x34fd600 thru 0x350461b (0x701c bytes) 2024-05-06T00:29:30.325 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:30.325 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:30.332 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:30.333+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:30.332 INFO:teuthology.orchestra.run.smithi186.stdout:5762 write 0xaf2ec00 thru 0xaf2f34b (0x74c bytes) 2024-05-06T00:29:30.333 INFO:teuthology.orchestra.run.smithi186.stdout:5763 write 0xd61a00 thru 0xd63358 (0x1959 bytes) 2024-05-06T00:29:30.333 INFO:teuthology.orchestra.run.smithi186.stdout:5764 read 0x6769600 thru 0x676e65c (0x505d bytes) 2024-05-06T00:29:30.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:30.358 INFO:teuthology.orchestra.run.smithi186.stdout:5765 write 0xb185000 thru 0xb18bd0c (0x6d0d bytes) 2024-05-06T00:29:30.359 INFO:teuthology.orchestra.run.smithi186.stdout:5767 write 0x397c00 thru 0x39f807 (0x7c08 bytes) 2024-05-06T00:29:30.390 INFO:teuthology.orchestra.run.smithi186.stdout:5768 write 0xca65800 thru 0xca6dbaa (0x83ab bytes) 2024-05-06T00:29:30.392 INFO:teuthology.orchestra.run.smithi186.stdout:5770 punch from 0x3532c00 to 0x353f800, (0xcc00 bytes) 2024-05-06T00:29:30.393 INFO:teuthology.orchestra.run.smithi186.stdout:5771 read 0xa0e0800 thru 0xa0ea705 (0x9f06 bytes) 2024-05-06T00:29:30.404 INFO:teuthology.orchestra.run.smithi186.stdout:5772 trunc from 0xca6dc00 to 0x6ded000 2024-05-06T00:29:30.433 INFO:teuthology.orchestra.run.smithi186.stdout:5775 read 0x3764e00 thru 0x3767374 (0x2575 bytes) 2024-05-06T00:29:30.433 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:30.433+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:30.451 INFO:teuthology.orchestra.run.smithi186.stdout:5776 write 0x711fa00 thru 0x712c9a2 (0xcfa3 bytes) 2024-05-06T00:29:30.453 INFO:teuthology.orchestra.run.smithi186.stdout:5778 read 0x607f600 thru 0x607ffcc (0x9cd bytes) 2024-05-06T00:29:30.453 INFO:teuthology.orchestra.run.smithi186.stdout:5782 read 0x534be00 thru 0x53508c1 (0x4ac2 bytes) 2024-05-06T00:29:30.475 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:30.475 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:30.491 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:30.492 INFO:teuthology.orchestra.run.smithi186.stdout:5785 write 0xe1adc00 thru 0xe1adf7c (0x37d bytes) 2024-05-06T00:29:30.492 INFO:teuthology.orchestra.run.smithi186.stdout:5788 write 0xaa7a400 thru 0xaa867c8 (0xc3c9 bytes) 2024-05-06T00:29:30.492 INFO:teuthology.orchestra.run.smithi186.stdout:5790 read 0xc3e3c00 thru 0xc3f29e9 (0xedea bytes) 2024-05-06T00:29:30.492 INFO:teuthology.orchestra.run.smithi186.stdout:5793 read 0x1d9200 thru 0x1df920 (0x6721 bytes) 2024-05-06T00:29:30.493 INFO:teuthology.orchestra.run.smithi186.stdout:5795 read 0x42cb000 thru 0x42d7a23 (0xca24 bytes) 2024-05-06T00:29:30.493 INFO:teuthology.orchestra.run.smithi186.stdout:5796 punch from 0x7b58800 to 0x7b5c800, (0x4000 bytes) 2024-05-06T00:29:30.494 INFO:teuthology.orchestra.run.smithi186.stdout:5798 read 0xc1c1600 thru 0xc1c7793 (0x6194 bytes) 2024-05-06T00:29:30.495 INFO:teuthology.orchestra.run.smithi186.stdout:5800 punch from 0x7442a00 to 0x744ca00, (0xa000 bytes) 2024-05-06T00:29:30.495 INFO:teuthology.orchestra.run.smithi186.stdout:5801 trunc from 0xe1ae000 to 0xdcdda00 2024-05-06T00:29:30.534 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:30.533+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:30.534 INFO:teuthology.orchestra.run.smithi186.stdout:5802 write 0xa700800 thru 0xa700e20 (0x621 bytes) 2024-05-06T00:29:30.537 INFO:teuthology.orchestra.run.smithi186.stdout:5803 read 0x4251e00 thru 0x425d958 (0xbb59 bytes) 2024-05-06T00:29:30.537 INFO:teuthology.orchestra.run.smithi186.stdout:5806 read 0xdb2600 thru 0xdbf82f (0xd230 bytes) 2024-05-06T00:29:30.537 INFO:teuthology.orchestra.run.smithi186.stdout:5809 read 0x6816800 thru 0x6823090 (0xc891 bytes) 2024-05-06T00:29:30.538 INFO:teuthology.orchestra.run.smithi186.stdout:5810 read 0xcf59c00 thru 0xcf613d3 (0x77d4 bytes) 2024-05-06T00:29:30.560 INFO:teuthology.orchestra.run.smithi186.stdout:5811 write 0xe3c0c00 thru 0xe3c20eb (0x14ec bytes) 2024-05-06T00:29:30.563 INFO:teuthology.orchestra.run.smithi186.stdout:5812 write 0xdf90400 thru 0xdf91159 (0xd5a bytes) 2024-05-06T00:29:30.563 INFO:teuthology.orchestra.run.smithi186.stdout:5813 write 0x1eea800 thru 0x1ef7f00 (0xd701 bytes) 2024-05-06T00:29:30.571 INFO:teuthology.orchestra.run.smithi186.stdout:5814 trunc from 0xe3c2200 to 0x9cf0c00 2024-05-06T00:29:30.607 INFO:teuthology.orchestra.run.smithi186.stdout:5815 read 0x4d33800 thru 0x4d36d6b (0x356c bytes) 2024-05-06T00:29:30.626 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:30.626 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:30.630 INFO:teuthology.orchestra.run.smithi186.stdout:5816 trunc from 0x9cf0c00 to 0xee3600 2024-05-06T00:29:30.634 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:30.634+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:30.642 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:30.684 INFO:teuthology.orchestra.run.smithi186.stdout:5817 write 0x11f1600 thru 0x11f1cba (0x6bb bytes) 2024-05-06T00:29:30.708 INFO:teuthology.orchestra.run.smithi186.stdout:5818 write 0x167a600 thru 0x167c5a3 (0x1fa4 bytes) 2024-05-06T00:29:30.709 INFO:teuthology.orchestra.run.smithi186.stdout:5821 read 0x42b800 thru 0x432c8c (0x748d bytes) 2024-05-06T00:29:30.709 INFO:teuthology.orchestra.run.smithi186.stdout:5823 read 0x689800 thru 0x6958c8 (0xc0c9 bytes) 2024-05-06T00:29:30.736 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:30.733+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:30.741 INFO:teuthology.orchestra.run.smithi186.stdout:5828 write 0x4ed3800 thru 0x4edce76 (0x9677 bytes) 2024-05-06T00:29:30.743 INFO:teuthology.orchestra.run.smithi186.stdout:5829 read 0xba2c00 thru 0xbaeaad (0xbeae bytes) 2024-05-06T00:29:30.779 INFO:teuthology.orchestra.run.smithi186.stdout:5830 write 0x833fe00 thru 0x834893e (0x8b3f bytes) 2024-05-06T00:29:30.780 INFO:teuthology.orchestra.run.smithi186.stdout:5833 read 0x78c2400 thru 0x78c55a4 (0x31a5 bytes) 2024-05-06T00:29:30.819 INFO:teuthology.orchestra.run.smithi186.stdout:5834 write 0xc3e4a00 thru 0xc3ee653 (0x9c54 bytes) 2024-05-06T00:29:30.820 INFO:teuthology.orchestra.run.smithi186.stdout:5835 write 0x5367000 thru 0x536c542 (0x5543 bytes) 2024-05-06T00:29:30.820 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:30.821 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:30.837 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:30.838+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:30.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:30.851 INFO:teuthology.orchestra.run.smithi186.stdout:5836 write 0xdbb1200 thru 0xdbb3721 (0x2522 bytes) 2024-05-06T00:29:30.853 INFO:teuthology.orchestra.run.smithi186.stdout:5838 punch from 0x9618a00 to 0x9623200, (0xa800 bytes) 2024-05-06T00:29:30.853 INFO:teuthology.orchestra.run.smithi186.stdout:5839 write 0x8d0600 thru 0x8d9aef (0x94f0 bytes) 2024-05-06T00:29:30.871 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:29:30.871 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:29:30.871 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-06T00:29:30.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-06T00:29:30.878 INFO:teuthology.orchestra.run.smithi186.stdout:5841 trunc from 0xdbb3800 to 0x3d83400 2024-05-06T00:29:30.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:29:30.911 INFO:teuthology.orchestra.run.smithi186.stdout:5842 write 0x9a5800 thru 0x9b29a3 (0xd1a4 bytes) 2024-05-06T00:29:30.915 INFO:teuthology.orchestra.run.smithi186.stdout:5843 trunc from 0x3d83400 to 0x4a54600 2024-05-06T00:29:30.938 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:30.937+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:30.970 INFO:teuthology.orchestra.run.smithi186.stdout:5844 write 0x96cba00 thru 0x96dad16 (0xf317 bytes) 2024-05-06T00:29:30.972 INFO:teuthology.orchestra.run.smithi186.stdout:5845 write 0x3bb7e00 thru 0x3bc0189 (0x838a bytes) 2024-05-06T00:29:30.976 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:30.976 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:30.995 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:31.002 INFO:teuthology.orchestra.run.smithi186.stdout:5846 write 0x9758600 thru 0x9764e8a (0xc88b bytes) 2024-05-06T00:29:31.004 INFO:teuthology.orchestra.run.smithi186.stdout:5847 read 0x5998e00 thru 0x59a537a (0xc57b bytes) 2024-05-06T00:29:31.037 INFO:teuthology.orchestra.run.smithi186.stdout:5849 write 0xda90a00 thru 0xda91187 (0x788 bytes) 2024-05-06T00:29:31.038 INFO:teuthology.orchestra.run.smithi186.stdout:5851 read 0x45d9800 thru 0x45e5057 (0xb858 bytes) 2024-05-06T00:29:31.039 INFO:teuthology.orchestra.run.smithi186.stdout:5852 write 0x4ef5600 thru 0x4f00299 (0xac9a bytes) 2024-05-06T00:29:31.039 INFO:teuthology.orchestra.run.smithi186.stdout:5853 punch from 0x1bbac00 to 0x1bc0800, (0x5c00 bytes) 2024-05-06T00:29:31.039 INFO:teuthology.orchestra.run.smithi186.stdout:5856 read 0x5331400 thru 0x533b2ff (0x9f00 bytes) 2024-05-06T00:29:31.039 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:31.038+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:31.070 INFO:teuthology.orchestra.run.smithi186.stdout:5858 write 0xeccc800 thru 0xecd068a (0x3e8b bytes) 2024-05-06T00:29:31.070 INFO:teuthology.orchestra.run.smithi186.stdout:5859 punch from 0x74bea00 to 0x74c0600, (0x1c00 bytes) 2024-05-06T00:29:31.090 INFO:teuthology.orchestra.run.smithi186.stdout:5860 trunc from 0xecd0800 to 0x49e1a00 2024-05-06T00:29:31.121 INFO:teuthology.orchestra.run.smithi186.stdout:5861 write 0x2778200 thru 0x27834d4 (0xb2d5 bytes) 2024-05-06T00:29:31.122 INFO:teuthology.orchestra.run.smithi186.stdout:5862 punch from 0x22b5400 to 0x22bf600, (0xa200 bytes) 2024-05-06T00:29:31.133 INFO:teuthology.orchestra.run.smithi186.stdout:5863 trunc from 0x49e1a00 to 0xa4d8000 2024-05-06T00:29:31.139 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:31.137+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:31.150 INFO:teuthology.orchestra.run.smithi186.stdout:5865 write 0x5eb7000 thru 0x5eba50b (0x350c bytes) 2024-05-06T00:29:31.151 INFO:teuthology.orchestra.run.smithi186.stdout:5867 read 0x6a87a00 thru 0x6a8d82c (0x5e2d bytes) 2024-05-06T00:29:31.156 INFO:teuthology.orchestra.run.smithi186.stdout:5868 trunc from 0xa4d8000 to 0x788ce00 2024-05-06T00:29:31.160 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:31.161 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:31.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:31.179 INFO:teuthology.orchestra.run.smithi186.stdout:5869 write 0x1938200 thru 0x1946ca3 (0xeaa4 bytes) 2024-05-06T00:29:31.181 INFO:teuthology.orchestra.run.smithi186.stdout:5873 punch from 0x3356a00 to 0x3363a00, (0xd000 bytes) 2024-05-06T00:29:31.183 INFO:teuthology.orchestra.run.smithi186.stdout:5877 flatten 2024-05-06T00:29:31.210 INFO:teuthology.orchestra.run.smithi186.stdout:5879 read 0x7423600 thru 0x7432d4c (0xf74d bytes) 2024-05-06T00:29:31.211 INFO:teuthology.orchestra.run.smithi186.stdout:5883 punch from 0x2588000 to 0x258d200, (0x5200 bytes) 2024-05-06T00:29:31.234 INFO:teuthology.orchestra.run.smithi186.stdout:5884 write 0x94afc00 thru 0x94b996d (0x9d6e bytes) 2024-05-06T00:29:31.235 INFO:teuthology.orchestra.run.smithi186.stdout:5885 read 0x5ab4a00 thru 0x5ab8ca2 (0x42a3 bytes) 2024-05-06T00:29:31.235 INFO:teuthology.orchestra.run.smithi186.stdout:5886 write 0x8385000 thru 0x838d459 (0x845a bytes) 2024-05-06T00:29:31.240 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:31.238+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:31.263 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 147, 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b583c614-b2ca-4769-90c0-44726bf69b34", 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-06T00:26:16.923985+0000", 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-06T00:29:28.851722+0000", 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-06T00:26:27.394947+0000", 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-06T00:28:54.626879+0000", 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5799936, 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-06T00:29:31.264 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 6, 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "luminous", 2024-05-06T00:29:31.265 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:28.436494+0000", 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:29:31.266 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-06T00:29:31.267 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "14", 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:29:31.268 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:29:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:29:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:29:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-06T00:29:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 5, 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 5, 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.40000000596046448, 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.61445111036300659, 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.74000000953674316, 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.61000001430511475 2024-05-06T00:29:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:33.941780+0000", 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:29:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-06T00:29:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:29:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:29:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:29:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:29:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:29:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 3.119999885559082, 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 3.119999885559082, 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 1.2000000476837158, 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 3.75, 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 3.75, 2024-05-06T00:29:31.281 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.61445111036300659, 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.74000000953674316, 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.61000001430511475 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:42.622595+0000", 2024-05-06T00:29:31.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:29:31.283 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:29:31.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:29:31.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-06T00:29:31.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-06T00:29:31.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-06T00:29:31.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-06T00:29:31.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-06T00:29:31.284 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:29:31.285 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:29:31.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:29:31.285 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:29:31.285 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:29:31.285 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:29:31.285 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:29:31.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.285 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "146", 2024-05-06T00:29:31.286 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:29:31.286 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:29:31.286 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:29:31.287 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:29:31.287 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:29:31.287 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 45, 2024-05-06T00:29:31.287 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 146, 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:29:31.288 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:29:31.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:29:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:29:31.291 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:29:31.291 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:29:31.291 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:29:31.291 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:29:31.291 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:29:31.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:29:31.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.292 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:29:31.292 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:29:31.292 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 5.5300002098083496, 2024-05-06T00:29:31.293 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 1.4099999666213989, 2024-05-06T00:29:31.293 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 1.2000000476837158, 2024-05-06T00:29:31.293 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 6.6399998664855957, 2024-05-06T00:29:31.294 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.690000057220459, 2024-05-06T00:29:31.294 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.61445111036300659, 2024-05-06T00:29:31.294 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.74000000953674316, 2024-05-06T00:29:31.294 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.61000001430511475, 2024-05-06T00:29:31.294 INFO:teuthology.orchestra.run.smithi078.stdout: "error_message": "pool 3 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d31b9a6a-864b-4c24-99db-8cc6cf3a5c24", 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:31.295 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 139, 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.296 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:31.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.300 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6817/1427564096", 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6819/1427564096", 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6823/1427564096", 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6821/1427564096", 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "86509826-0e53-4678-8260-17c97c05af54", 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0.9229278564453125, 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 144, 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:31.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.305 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6803/2663808859", 2024-05-06T00:29:31.307 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6807/2663808859", 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/2663808859", 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6811/2663808859", 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:29:31.308 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "8973b524-521e-41c0-87f2-85cb390fbfeb", 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.2727203369140625, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 127, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.310 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:31.312 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6802/385938386", 2024-05-06T00:29:31.313 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6806/385938386", 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6814/385938386", 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6810/385938386", 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "3b0d07bd-132e-48ff-83ac-7182eee05df9", 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:31.314 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0, 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 139, 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6811", 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:31.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6812", 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6814", 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:31.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6816", 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6821", 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:31.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6822", 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6817", 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:31.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6819", 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6812/4178038317", 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6816/4178038317", 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6822/4178038317", 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6819/4178038317", 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:31.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2", 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.833282470703125, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 132, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:31.320 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6808", 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6809", 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.321 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6810", 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6813", 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.322 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6820", 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6823", 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:31.323 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6815", 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6818", 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6809/1052555350", 2024-05-06T00:29:31.324 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6813/1052555350", 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1052555350", 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6818/1052555350", 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "34b61f5f-e407-4414-8bc5-11263bbe261a", 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.5807037353515625, 2024-05-06T00:29:31.325 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 135, 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6800", 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.326 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6801", 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6802", 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.327 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6803", 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6806", 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6807", 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6804", 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:29:31.329 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6805", 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6801/2019146845", 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6803/2019146845", 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6807/2019146845", 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6805/2019146845", 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.330 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.276988+0000", 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:29:31.331 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.054473+0000", 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:31.332 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.154558+0000", 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.284746+0000", 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.259847+0000", 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:29:31.334 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.180375+0000", 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [ 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.15", 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: 4, 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.335 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [ 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "2.1", 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 2, 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 3 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.6", 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-06T00:29:31.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 1, 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 4 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [], 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-06T00:29:31.337 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [], 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [], 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-06T00:29:31.338 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-06T00:29:31.339 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-06T00:29:31.339 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-06T00:29:31.339 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-06T00:29:31.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:31.339 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:31.341 INFO:teuthology.orchestra.run.smithi186.stdout:5890 write 0xbe09e00 thru 0xbe17340 (0xd541 bytes) 2024-05-06T00:29:31.341 INFO:teuthology.orchestra.run.smithi186.stdout:5892 read 0xa780c00 thru 0xa783c43 (0x3044 bytes) 2024-05-06T00:29:31.341 INFO:teuthology.orchestra.run.smithi186.stdout:5894 trunc from 0xbe17400 to 0xa247800 2024-05-06T00:29:31.341 INFO:teuthology.orchestra.run.smithi186.stdout:5896 read 0x38e200 thru 0x3924fc (0x42fd bytes) 2024-05-06T00:29:31.341 INFO:teuthology.orchestra.run.smithi186.stdout:5897 trunc from 0xa247800 to 0xdcb4c00 2024-05-06T00:29:31.341 INFO:teuthology.orchestra.run.smithi186.stdout:5898 clone 43 order 19 su 65536 sc 4 2024-05-06T00:29:31.342 INFO:tasks.thrashosds.thrasher:j is {'epoch': 147, 'fsid': 'b583c614-b2ca-4769-90c0-44726bf69b34', 'created': '2024-05-06T00:26:16.923985+0000', 'modified': '2024-05-06T00:29:28.851722+0000', 'last_up_change': '2024-05-06T00:26:27.394947+0000', 'last_in_change': '2024-05-06T00:28:54.626879+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T00:26:28.436494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5, 'score_stable': 5, 'optimal_score': 0.4000000059604645, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6144511103630066, 'average_primary_affinity': 0.7400000095367432, 'average_primary_affinity_weighted': 0.6100000143051147}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-05-06T00:26:33.941780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.119999885559082, 'score_stable': 3.119999885559082, 'optimal_score': 1.2000000476837158, 'raw_score_acting': 3.75, 'raw_score_stable': 3.75, 'primary_affinity_weighted': 0.6144511103630066, 'average_primary_affinity': 0.7400000095367432, 'average_primary_affinity_weighted': 0.6100000143051147}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-06T00:26:42.622595+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 45, 'snap_epoch': 146, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.53000020980835, 'score_stable': 1.409999966621399, 'optimal_score': 1.2000000476837158, 'raw_score_acting': 6.639999866485596, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.6144511103630066, 'average_primary_affinity': 0.7400000095367432, 'average_primary_affinity_weighted': 0.6100000143051147, 'error_message': 'pool 3 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': 'd31b9a6a-864b-4c24-99db-8cc6cf3a5c24', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 139, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6817', 'nonce': 1427564096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6818', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 1427564096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6822', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 1427564096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6821', 'nonce': 1427564096}]}, 'public_addr': '172.21.15.78:6817/1427564096', 'cluster_addr': '172.21.15.78:6819/1427564096', 'heartbeat_back_addr': '172.21.15.78:6823/1427564096', 'heartbeat_front_addr': '172.21.15.78:6821/1427564096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '86509826-0e53-4678-8260-17c97c05af54', 'up': 1, 'in': 1, 'weight': 0.9229278564453125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6801', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6803', 'nonce': 2663808859}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6805', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6807', 'nonce': 2663808859}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6813', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 2663808859}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6809', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6811', 'nonce': 2663808859}]}, 'public_addr': '172.21.15.78:6803/2663808859', 'cluster_addr': '172.21.15.78:6807/2663808859', 'heartbeat_back_addr': '172.21.15.78:6815/2663808859', 'heartbeat_front_addr': '172.21.15.78:6811/2663808859', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8973b524-521e-41c0-87f2-85cb390fbfeb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2727203369140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6802', 'nonce': 385938386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6804', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6806', 'nonce': 385938386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6812', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6814', 'nonce': 385938386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6808', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6810', 'nonce': 385938386}]}, 'public_addr': '172.21.15.78:6802/385938386', 'cluster_addr': '172.21.15.78:6806/385938386', 'heartbeat_back_addr': '172.21.15.78:6814/385938386', 'heartbeat_front_addr': '172.21.15.78:6810/385938386', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3b0d07bd-132e-48ff-83ac-7182eee05df9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 139, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6811', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6812', 'nonce': 4178038317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6816', 'nonce': 4178038317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6821', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6822', 'nonce': 4178038317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6817', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 4178038317}]}, 'public_addr': '172.21.15.146:6812/4178038317', 'cluster_addr': '172.21.15.146:6816/4178038317', 'heartbeat_back_addr': '172.21.15.146:6822/4178038317', 'heartbeat_front_addr': '172.21.15.146:6819/4178038317', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.833282470703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 1052555350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 1052555350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1052555350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6815', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6818', 'nonce': 1052555350}]}, 'public_addr': '172.21.15.146:6809/1052555350', 'cluster_addr': '172.21.15.146:6813/1052555350', 'heartbeat_back_addr': '172.21.15.146:6823/1052555350', 'heartbeat_front_addr': '172.21.15.146:6818/1052555350', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '34b61f5f-e407-4414-8bc5-11263bbe261a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5807037353515625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 2019146845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 2019146845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 2019146845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 2019146845}]}, 'public_addr': '172.21.15.146:6801/2019146845', 'cluster_addr': '172.21.15.146:6803/2019146845', 'heartbeat_back_addr': '172.21.15.146:6807/2019146845', 'heartbeat_front_addr': '172.21.15.146:6805/2019146845', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.276988+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.054473+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.154558+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.284746+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.259847+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.180375+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.15', 'osds': [4, 1]}], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 2, 'to': 3}]}, {'pgid': '3.6', 'mappings': [{'from': 1, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-06T00:29:31.342 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:29:31.343 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:31.338+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:31.343 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:31.343 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:31.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:31.441 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:31.442+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:31.519 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:31.519 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:31.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:31.542 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:31.542+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:29:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:29:31.642 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:31.641+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:31.652 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:31.652 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:31.656 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-05-06T00:29:30.439417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1306054587,"num_objects":2126,"num_object_clones":502,"num_object_copies":4252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":1030,"num_read":237808,"num_read_kb":786209,"num_write":16720,"num_write_kb":184764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5554,"num_bytes_recovered":74299906,"num_keys_recovered":904,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15409,"ondisk_log_size":15409,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":481176,"kb_used_data":148368,"kb_used_omap":9,"kb_used_meta":332726,"kb_avail":565749864,"statfs":{"total":579820584960,"available":579327860736,"internally_reserved":0,"allocated":151928832,"data_stored":180727592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":340712096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":55054454,"num_objects":77,"num_object_clones":30,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":25,"num_read":11105,"num_read_kb":68156,"num_write":886,"num_write_kb":14353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.596908"},"pg_stats":[{"pgid":"3.1f","version":"145'532","reported_seq":12107,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.504239+0000","last_change":"2024-05-06T00:28:31.628317+0000","last_active":"2024-05-06T00:29:27.504239+0000","last_peered":"2024-05-06T00:29:27.504239+0000","last_clean":"2024-05-06T00:29:27.504239+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:29:27.504239+0000","last_undegraded":"2024-05-06T00:29:27.504239+0000","last_fullsized":"2024-05-06T00:29:27.504239+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:35:39.995789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067141100000000001,"stat_sum":{"num_bytes":29662343,"num_objects":66,"num_object_clones":15,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":33,"num_read":11431,"num_read_kb":39030,"num_write":445,"num_write_kb":2488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"144'280","reported_seq":9373,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.844179+0000","last_change":"2024-05-06T00:28:58.676392+0000","last_active":"2024-05-06T00:29:26.844179+0000","last_peered":"2024-05-06T00:29:26.844179+0000","last_clean":"2024-05-06T00:29:26.844179+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:29:26.844179+0000","last_undegraded":"2024-05-06T00:29:26.844179+0000","last_fullsized":"2024-05-06T00:29:26.844179+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:04:22.710840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059673699999999996,"stat_sum":{"num_bytes":77692488,"num_objects":78,"num_object_clones":24,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":31,"num_read":8597,"num_read_kb":47496,"num_write":251,"num_write_kb":17529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"142'377","reported_seq":3270,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.872985+0000","last_change":"2024-05-06T00:29:13.787230+0000","last_active":"2024-05-06T00:29:26.872985+0000","last_peered":"2024-05-06T00:29:26.872985+0000","last_clean":"2024-05-06T00:29:26.872985+0000","last_became_active":"2024-05-06T00:29:13.784483+0000","last_became_peered":"2024-05-06T00:29:13.784483+0000","last_unstale":"2024-05-06T00:29:26.872985+0000","last_undegraded":"2024-05-06T00:29:26.872985+0000","last_fullsized":"2024-05-06T00:29:26.872985+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:20:50.071503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039913359999999998,"stat_sum":{"num_bytes":47189556,"num_objects":58,"num_object_clones":13,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":36,"num_read":3128,"num_read_kb":11594,"num_write":353,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"145'703","reported_seq":8310,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.520169+0000","last_change":"2024-05-06T00:29:07.739802+0000","last_active":"2024-05-06T00:29:27.520169+0000","last_peered":"2024-05-06T00:29:27.520169+0000","last_clean":"2024-05-06T00:29:27.520169+0000","last_became_active":"2024-05-06T00:29:07.735705+0000","last_became_peered":"2024-05-06T00:29:07.735705+0000","last_unstale":"2024-05-06T00:29:27.520169+0000","last_undegraded":"2024-05-06T00:29:27.520169+0000","last_fullsized":"2024-05-06T00:29:27.520169+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:05:12.524767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033200619999999999,"stat_sum":{"num_bytes":46458143,"num_objects":71,"num_object_clones":19,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":29,"num_read":11078,"num_read_kb":29958,"num_write":612,"num_write_kb":5152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"145'991","reported_seq":9078,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.270612+0000","last_change":"2024-05-06T00:28:05.824053+0000","last_active":"2024-05-06T00:29:27.270612+0000","last_peered":"2024-05-06T00:29:27.270612+0000","last_clean":"2024-05-06T00:29:27.270612+0000","last_became_active":"2024-05-06T00:28:05.820651+0000","last_became_peered":"2024-05-06T00:28:05.820651+0000","last_unstale":"2024-05-06T00:29:27.270612+0000","last_undegraded":"2024-05-06T00:29:27.270612+0000","last_fullsized":"2024-05-06T00:29:27.270612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:35:21.073064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093508700000000003,"stat_sum":{"num_bytes":60875901,"num_objects":94,"num_object_clones":27,"num_object_copies":188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":44,"num_read":12475,"num_read_kb":32546,"num_write":999,"num_write_kb":8053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"145'325","reported_seq":6022,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.503046+0000","last_change":"2024-05-06T00:26:45.721833+0000","last_active":"2024-05-06T00:29:27.503046+0000","last_peered":"2024-05-06T00:29:27.503046+0000","last_clean":"2024-05-06T00:29:27.503046+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:29:27.503046+0000","last_undegraded":"2024-05-06T00:29:27.503046+0000","last_fullsized":"2024-05-06T00:29:27.503046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:22:42.027279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024202849999999999,"stat_sum":{"num_bytes":38636263,"num_objects":59,"num_object_clones":11,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":28,"num_read":6287,"num_read_kb":23020,"num_write":357,"num_write_kb":13596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"145'586","reported_seq":3300,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.331611+0000","last_change":"2024-05-06T00:26:45.725095+0000","last_active":"2024-05-06T00:29:27.331611+0000","last_peered":"2024-05-06T00:29:27.331611+0000","last_clean":"2024-05-06T00:29:27.331611+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:29:27.331611+0000","last_undegraded":"2024-05-06T00:29:27.331611+0000","last_fullsized":"2024-05-06T00:29:27.331611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:44:00.839803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043183499999999999,"stat_sum":{"num_bytes":22327754,"num_objects":65,"num_object_clones":17,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":27,"num_read":4876,"num_read_kb":16552,"num_write":764,"num_write_kb":5613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"145'595","reported_seq":13821,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.912825+0000","last_change":"2024-05-06T00:29:19.817685+0000","last_active":"2024-05-06T00:29:26.912825+0000","last_peered":"2024-05-06T00:29:26.912825+0000","last_clean":"2024-05-06T00:29:26.912825+0000","last_became_active":"2024-05-06T00:29:19.816383+0000","last_became_peered":"2024-05-06T00:29:19.816383+0000","last_unstale":"2024-05-06T00:29:26.912825+0000","last_undegraded":"2024-05-06T00:29:26.912825+0000","last_fullsized":"2024-05-06T00:29:26.912825+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:16:12.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086199499999999995,"stat_sum":{"num_bytes":29062725,"num_objects":66,"num_object_clones":19,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":30,"num_read":13505,"num_read_kb":24269,"num_write":429,"num_write_kb":3272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"141'277","reported_seq":4378,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.849164+0000","last_change":"2024-05-06T00:28:25.971398+0000","last_active":"2024-05-06T00:29:26.849164+0000","last_peered":"2024-05-06T00:29:26.849164+0000","last_clean":"2024-05-06T00:29:26.849164+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:29:26.849164+0000","last_undegraded":"2024-05-06T00:29:26.849164+0000","last_fullsized":"2024-05-06T00:29:26.849164+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:25:37.382519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010871520000000001,"stat_sum":{"num_bytes":81067619,"num_objects":66,"num_object_clones":15,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":34,"num_read":3505,"num_read_kb":18360,"num_write":252,"num_write_kb":5016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"142'432","reported_seq":2418,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.872831+0000","last_change":"2024-05-06T00:28:31.621968+0000","last_active":"2024-05-06T00:29:26.872831+0000","last_peered":"2024-05-06T00:29:26.872831+0000","last_clean":"2024-05-06T00:29:26.872831+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:29:26.872831+0000","last_undegraded":"2024-05-06T00:29:26.872831+0000","last_fullsized":"2024-05-06T00:29:26.872831+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:02:55.743530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040656920000000001,"stat_sum":{"num_bytes":52991079,"num_objects":63,"num_object_clones":18,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":32,"num_read":2599,"num_read_kb":14330,"num_write":562,"num_write_kb":1821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"145'238","reported_seq":5108,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.857492+0000","last_change":"2024-05-06T00:26:51.611424+0000","last_active":"2024-05-06T00:29:26.857492+0000","last_peered":"2024-05-06T00:29:26.857492+0000","last_clean":"2024-05-06T00:29:26.857492+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:29:26.857492+0000","last_undegraded":"2024-05-06T00:29:26.857492+0000","last_fullsized":"2024-05-06T00:29:26.857492+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:55:34.912021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16366630,"num_objects":52,"num_object_clones":15,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":28,"num_read":4782,"num_read_kb":10538,"num_write":232,"num_write_kb":15281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"145'436","reported_seq":7074,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.073350+0000","last_change":"2024-05-06T00:28:31.621451+0000","last_active":"2024-05-06T00:29:27.073350+0000","last_peered":"2024-05-06T00:29:27.073350+0000","last_clean":"2024-05-06T00:29:27.073350+0000","last_became_active":"2024-05-06T00:28:31.620668+0000","last_became_peered":"2024-05-06T00:28:31.620668+0000","last_unstale":"2024-05-06T00:29:27.073350+0000","last_undegraded":"2024-05-06T00:29:27.073350+0000","last_fullsized":"2024-05-06T00:29:27.073350+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:01:06.415791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011408639999999999,"stat_sum":{"num_bytes":32230800,"num_objects":55,"num_object_clones":11,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":27,"num_read":7361,"num_read_kb":20562,"num_write":595,"num_write_kb":2391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":217,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.857151+0000","last_change":"2024-05-06T00:28:27.989114+0000","last_active":"2024-05-06T00:29:26.857151+0000","last_peered":"2024-05-06T00:29:26.857151+0000","last_clean":"2024-05-06T00:29:26.857151+0000","last_became_active":"2024-05-06T00:28:27.988260+0000","last_became_peered":"2024-05-06T00:28:27.988260+0000","last_unstale":"2024-05-06T00:29:26.857151+0000","last_undegraded":"2024-05-06T00:29:26.857151+0000","last_fullsized":"2024-05-06T00:29:26.857151+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:19:42.564014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"145'818","reported_seq":7107,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.869462+0000","last_change":"2024-05-06T00:28:06.817938+0000","last_active":"2024-05-06T00:29:26.869462+0000","last_peered":"2024-05-06T00:29:26.869462+0000","last_clean":"2024-05-06T00:29:26.869462+0000","last_became_active":"2024-05-06T00:28:06.816588+0000","last_became_peered":"2024-05-06T00:28:06.816588+0000","last_unstale":"2024-05-06T00:29:26.869462+0000","last_undegraded":"2024-05-06T00:29:26.869462+0000","last_fullsized":"2024-05-06T00:29:26.869462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:29:04.174478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048288900000000002,"stat_sum":{"num_bytes":77912710,"num_objects":63,"num_object_clones":14,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":26,"num_read":10309,"num_read_kb":28826,"num_write":1029,"num_write_kb":7287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":279,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.845420+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:29:26.845420+0000","last_peered":"2024-05-06T00:29:26.845420+0000","last_clean":"2024-05-06T00:29:26.845420+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:29:26.845420+0000","last_undegraded":"2024-05-06T00:29:26.845420+0000","last_fullsized":"2024-05-06T00:29:26.845420+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"142'636","reported_seq":6071,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.845760+0000","last_change":"2024-05-06T00:29:19.818357+0000","last_active":"2024-05-06T00:29:26.845760+0000","last_peered":"2024-05-06T00:29:26.845760+0000","last_clean":"2024-05-06T00:29:26.845760+0000","last_became_active":"2024-05-06T00:29:19.817230+0000","last_became_peered":"2024-05-06T00:29:19.817230+0000","last_unstale":"2024-05-06T00:29:26.845760+0000","last_undegraded":"2024-05-06T00:29:26.845760+0000","last_fullsized":"2024-05-06T00:29:26.845760+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:11:34.828343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043687199999999999,"stat_sum":{"num_bytes":29721735,"num_objects":62,"num_object_clones":16,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":29,"num_read":8364,"num_read_kb":19346,"num_write":625,"num_write_kb":2719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":148,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.848915+0000","last_change":"2024-05-06T00:28:27.997043+0000","last_active":"2024-05-06T00:29:26.848915+0000","last_peered":"2024-05-06T00:29:26.848915+0000","last_clean":"2024-05-06T00:29:26.848915+0000","last_became_active":"2024-05-06T00:28:27.996051+0000","last_became_peered":"2024-05-06T00:28:27.996051+0000","last_unstale":"2024-05-06T00:29:26.848915+0000","last_undegraded":"2024-05-06T00:29:26.848915+0000","last_fullsized":"2024-05-06T00:29:26.848915+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:03:48.195523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"142'527","reported_seq":8763,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.845508+0000","last_change":"2024-05-06T00:28:59.669820+0000","last_active":"2024-05-06T00:29:26.845508+0000","last_peered":"2024-05-06T00:29:26.845508+0000","last_clean":"2024-05-06T00:29:26.845508+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:29:26.845508+0000","last_undegraded":"2024-05-06T00:29:26.845508+0000","last_fullsized":"2024-05-06T00:29:26.845508+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:04:45.808680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039298300000000002,"stat_sum":{"num_bytes":51007617,"num_objects":77,"num_object_clones":15,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":37,"num_read":9529,"num_read_kb":30473,"num_write":605,"num_write_kb":4922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":279,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.845645+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:29:26.845645+0000","last_peered":"2024-05-06T00:29:26.845645+0000","last_clean":"2024-05-06T00:29:26.845645+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:29:26.845645+0000","last_undegraded":"2024-05-06T00:29:26.845645+0000","last_fullsized":"2024-05-06T00:29:26.845645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"145'318","reported_seq":2945,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.321205+0000","last_change":"2024-05-06T00:28:58.680125+0000","last_active":"2024-05-06T00:29:27.321205+0000","last_peered":"2024-05-06T00:29:27.321205+0000","last_clean":"2024-05-06T00:29:27.321205+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:29:27.321205+0000","last_undegraded":"2024-05-06T00:29:27.321205+0000","last_fullsized":"2024-05-06T00:29:27.321205+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:13:25.497799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040487759999999996,"stat_sum":{"num_bytes":22951035,"num_objects":46,"num_object_clones":9,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":22,"num_read":2860,"num_read_kb":9928,"num_write":336,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"143'280","reported_seq":3080,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.857174+0000","last_change":"2024-05-06T00:28:02.884939+0000","last_active":"2024-05-06T00:29:26.857174+0000","last_peered":"2024-05-06T00:29:26.857174+0000","last_clean":"2024-05-06T00:29:26.857174+0000","last_became_active":"2024-05-06T00:28:02.782051+0000","last_became_peered":"2024-05-06T00:28:02.782051+0000","last_unstale":"2024-05-06T00:29:26.857174+0000","last_undegraded":"2024-05-06T00:29:26.857174+0000","last_fullsized":"2024-05-06T00:29:26.857174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:09:32.928575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083824599999999996,"stat_sum":{"num_bytes":29403824,"num_objects":60,"num_object_clones":15,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":32,"num_read":2800,"num_read_kb":17254,"num_write":317,"num_write_kb":1934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.845671+0000","last_change":"2024-05-06T00:29:07.737006+0000","last_active":"2024-05-06T00:29:26.845671+0000","last_peered":"2024-05-06T00:29:26.845671+0000","last_clean":"2024-05-06T00:29:26.845671+0000","last_became_active":"2024-05-06T00:29:07.736607+0000","last_became_peered":"2024-05-06T00:29:07.736607+0000","last_unstale":"2024-05-06T00:29:26.845671+0000","last_undegraded":"2024-05-06T00:29:26.845671+0000","last_fullsized":"2024-05-06T00:29:26.845671+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:33:39.712007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"145'531","reported_seq":6029,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.883211+0000","last_change":"2024-05-06T00:28:59.669906+0000","last_active":"2024-05-06T00:29:26.883211+0000","last_peered":"2024-05-06T00:29:26.883211+0000","last_clean":"2024-05-06T00:29:26.883211+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:29:26.883211+0000","last_undegraded":"2024-05-06T00:29:26.883211+0000","last_fullsized":"2024-05-06T00:29:26.883211+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:09:46.893739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185000000000005,"stat_sum":{"num_bytes":42837679,"num_objects":61,"num_object_clones":12,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":27,"num_read":6355,"num_read_kb":23172,"num_write":676,"num_write_kb":4074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.848406+0000","last_change":"2024-05-06T00:29:26.848406+0000","last_active":"2024-05-06T00:29:26.848406+0000","last_peered":"2024-05-06T00:29:26.848406+0000","last_clean":"2024-05-06T00:29:26.848406+0000","last_became_active":"2024-05-06T00:29:26.847624+0000","last_became_peered":"2024-05-06T00:29:26.847624+0000","last_unstale":"2024-05-06T00:29:26.848406+0000","last_undegraded":"2024-05-06T00:29:26.848406+0000","last_fullsized":"2024-05-06T00:29:26.848406+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:37:34.416843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"143'289","reported_seq":4437,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.848433+0000","last_change":"2024-05-06T00:28:05.821874+0000","last_active":"2024-05-06T00:29:26.848433+0000","last_peered":"2024-05-06T00:29:26.848433+0000","last_clean":"2024-05-06T00:29:26.848433+0000","last_became_active":"2024-05-06T00:28:05.820367+0000","last_became_peered":"2024-05-06T00:28:05.820367+0000","last_unstale":"2024-05-06T00:29:26.848433+0000","last_undegraded":"2024-05-06T00:29:26.848433+0000","last_fullsized":"2024-05-06T00:29:26.848433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:25:39.785294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052917899999999996,"stat_sum":{"num_bytes":46107182,"num_objects":65,"num_object_clones":16,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":29,"num_read":3900,"num_read_kb":12779,"num_write":306,"num_write_kb":2195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":218,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.846236+0000","last_change":"2024-05-06T00:28:27.993179+0000","last_active":"2024-05-06T00:29:26.846236+0000","last_peered":"2024-05-06T00:29:26.846236+0000","last_clean":"2024-05-06T00:29:26.846236+0000","last_became_active":"2024-05-06T00:28:27.991961+0000","last_became_peered":"2024-05-06T00:28:27.991961+0000","last_unstale":"2024-05-06T00:29:26.846236+0000","last_undegraded":"2024-05-06T00:29:26.846236+0000","last_fullsized":"2024-05-06T00:29:26.846236+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:21:51.936053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"145'399","reported_seq":9454,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.079049+0000","last_change":"2024-05-06T00:28:25.970874+0000","last_active":"2024-05-06T00:29:27.079049+0000","last_peered":"2024-05-06T00:29:27.079049+0000","last_clean":"2024-05-06T00:29:27.079049+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:29:27.079049+0000","last_undegraded":"2024-05-06T00:29:27.079049+0000","last_fullsized":"2024-05-06T00:29:27.079049+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:31:02.376319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035357200000000002,"stat_sum":{"num_bytes":21911193,"num_objects":52,"num_object_clones":11,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":26,"num_read":9939,"num_read_kb":22904,"num_write":315,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.872728+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:29:26.872728+0000","last_peered":"2024-05-06T00:29:26.872728+0000","last_clean":"2024-05-06T00:29:26.872728+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:29:26.872728+0000","last_undegraded":"2024-05-06T00:29:26.872728+0000","last_fullsized":"2024-05-06T00:29:26.872728+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":166,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:26.848473+0000","last_change":"2024-05-06T00:28:28.005137+0000","last_active":"2024-05-06T00:29:26.848473+0000","last_peered":"2024-05-06T00:29:26.848473+0000","last_clean":"2024-05-06T00:29:26.848473+0000","last_became_active":"2024-05-06T00:28:27.995351+0000","last_became_peered":"2024-05-06T00:28:27.995351+0000","last_unstale":"2024-05-06T00:29:26.848473+0000","last_undegraded":"2024-05-06T00:29:26.848473+0000","last_fullsized":"2024-05-06T00:29:26.848473+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:17:37.661069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"145'441","reported_seq":4518,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.520381+0000","last_change":"2024-05-06T00:29:19.819457+0000","last_active":"2024-05-06T00:29:27.520381+0000","last_peered":"2024-05-06T00:29:27.520381+0000","last_clean":"2024-05-06T00:29:27.520381+0000","last_became_active":"2024-05-06T00:29:19.818623+0000","last_became_peered":"2024-05-06T00:29:19.818623+0000","last_unstale":"2024-05-06T00:29:27.520381+0000","last_undegraded":"2024-05-06T00:29:27.520381+0000","last_fullsized":"2024-05-06T00:29:27.520381+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:21:31.013734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00134081,"stat_sum":{"num_bytes":33908454,"num_objects":80,"num_object_clones":20,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":41,"num_read":4724,"num_read_kb":21639,"num_write":507,"num_write_kb":7494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"145'485","reported_seq":5424,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.071727+0000","last_change":"2024-05-06T00:28:58.674634+0000","last_active":"2024-05-06T00:29:27.071727+0000","last_peered":"2024-05-06T00:29:27.071727+0000","last_clean":"2024-05-06T00:29:27.071727+0000","last_became_active":"2024-05-06T00:28:58.673289+0000","last_became_peered":"2024-05-06T00:28:58.673289+0000","last_unstale":"2024-05-06T00:29:27.071727+0000","last_undegraded":"2024-05-06T00:29:27.071727+0000","last_fullsized":"2024-05-06T00:29:27.071727+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:33:57.625184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021309100000000002,"stat_sum":{"num_bytes":35933484,"num_objects":72,"num_object_clones":17,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":35,"num_read":5424,"num_read_kb":27597,"num_write":630,"num_write_kb":13140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"145'630","reported_seq":7122,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.517620+0000","last_change":"2024-05-06T00:28:32.628240+0000","last_active":"2024-05-06T00:29:27.517620+0000","last_peered":"2024-05-06T00:29:27.517620+0000","last_clean":"2024-05-06T00:29:27.517620+0000","last_became_active":"2024-05-06T00:28:32.626303+0000","last_became_peered":"2024-05-06T00:28:32.626303+0000","last_unstale":"2024-05-06T00:29:27.517620+0000","last_undegraded":"2024-05-06T00:29:27.517620+0000","last_fullsized":"2024-05-06T00:29:27.517620+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:52:33.118806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011660139999999999,"stat_sum":{"num_bytes":49293689,"num_objects":88,"num_object_clones":20,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":41,"num_read":8206,"num_read_kb":39330,"num_write":623,"num_write_kb":8141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"145'282","reported_seq":6495,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.270595+0000","last_change":"2024-05-06T00:28:06.813891+0000","last_active":"2024-05-06T00:29:27.270595+0000","last_peered":"2024-05-06T00:29:27.270595+0000","last_clean":"2024-05-06T00:29:27.270595+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:29:27.270595+0000","last_undegraded":"2024-05-06T00:29:27.270595+0000","last_fullsized":"2024-05-06T00:29:27.270595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:31:15.629277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062030799999999997,"stat_sum":{"num_bytes":35207450,"num_objects":57,"num_object_clones":13,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":30,"num_read":4771,"num_read_kb":20788,"num_write":306,"num_write_kb":6654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"145'345","reported_seq":6287,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.516639+0000","last_change":"2024-05-06T00:29:07.736666+0000","last_active":"2024-05-06T00:29:27.516639+0000","last_peered":"2024-05-06T00:29:27.516639+0000","last_clean":"2024-05-06T00:29:27.516639+0000","last_became_active":"2024-05-06T00:29:07.734795+0000","last_became_peered":"2024-05-06T00:29:07.734795+0000","last_unstale":"2024-05-06T00:29:27.516639+0000","last_undegraded":"2024-05-06T00:29:27.516639+0000","last_fullsized":"2024-05-06T00:29:27.516639+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:06:59.623383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003830848,"stat_sum":{"num_bytes":35500630,"num_objects":71,"num_object_clones":15,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":33,"num_read":5808,"num_read_kb":51951,"num_write":331,"num_write_kb":8900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"145'780","reported_seq":10502,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.074634+0000","last_change":"2024-05-06T00:29:19.813529+0000","last_active":"2024-05-06T00:29:27.074634+0000","last_peered":"2024-05-06T00:29:27.074634+0000","last_clean":"2024-05-06T00:29:27.074634+0000","last_became_active":"2024-05-06T00:29:19.811713+0000","last_became_peered":"2024-05-06T00:29:19.811713+0000","last_unstale":"2024-05-06T00:29:27.074634+0000","last_undegraded":"2024-05-06T00:29:27.074634+0000","last_fullsized":"2024-05-06T00:29:27.074634+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:15:39.648195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002015387,"stat_sum":{"num_bytes":27467865,"num_objects":79,"num_object_clones":19,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":44,"num_read":11609,"num_read_kb":25469,"num_write":1091,"num_write_kb":4066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"145'718","reported_seq":11868,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.070236+0000","last_change":"2024-05-06T00:29:07.737933+0000","last_active":"2024-05-06T00:29:27.070236+0000","last_peered":"2024-05-06T00:29:27.070236+0000","last_clean":"2024-05-06T00:29:27.070236+0000","last_became_active":"2024-05-06T00:29:07.735487+0000","last_became_peered":"2024-05-06T00:29:07.735487+0000","last_unstale":"2024-05-06T00:29:27.070236+0000","last_undegraded":"2024-05-06T00:29:27.070236+0000","last_fullsized":"2024-05-06T00:29:27.070236+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:44:39.696560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042574209999999999,"stat_sum":{"num_bytes":41599667,"num_objects":80,"num_object_clones":20,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":38,"num_read":14494,"num_read_kb":28355,"num_write":953,"num_write_kb":4397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"145'289","reported_seq":3722,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.463600+0000","last_change":"2024-05-06T00:28:32.630504+0000","last_active":"2024-05-06T00:29:27.463600+0000","last_peered":"2024-05-06T00:29:27.463600+0000","last_clean":"2024-05-06T00:29:27.463600+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:29:27.463600+0000","last_undegraded":"2024-05-06T00:29:27.463600+0000","last_fullsized":"2024-05-06T00:29:27.463600+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:29:01.161944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040729959999999997,"stat_sum":{"num_bytes":16374323,"num_objects":52,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":26,"num_read":3331,"num_read_kb":17877,"num_write":339,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"145'625","reported_seq":10917,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.072391+0000","last_change":"2024-05-06T00:29:07.736408+0000","last_active":"2024-05-06T00:29:27.072391+0000","last_peered":"2024-05-06T00:29:27.072391+0000","last_clean":"2024-05-06T00:29:27.072391+0000","last_became_active":"2024-05-06T00:29:07.734992+0000","last_became_peered":"2024-05-06T00:29:07.734992+0000","last_unstale":"2024-05-06T00:29:27.072391+0000","last_undegraded":"2024-05-06T00:29:27.072391+0000","last_fullsized":"2024-05-06T00:29:27.072391+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:34:12.730462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032969919999999999,"stat_sum":{"num_bytes":39411780,"num_objects":63,"num_object_clones":11,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":30,"num_read":12516,"num_read_kb":30264,"num_write":610,"num_write_kb":4427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"145'361","reported_seq":5395,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.070890+0000","last_change":"2024-05-06T00:28:31.037532+0000","last_active":"2024-05-06T00:29:27.070890+0000","last_peered":"2024-05-06T00:29:27.070890+0000","last_clean":"2024-05-06T00:29:27.070890+0000","last_became_active":"2024-05-06T00:28:31.036279+0000","last_became_peered":"2024-05-06T00:28:31.036279+0000","last_unstale":"2024-05-06T00:29:27.070890+0000","last_undegraded":"2024-05-06T00:29:27.070890+0000","last_fullsized":"2024-05-06T00:29:27.070890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:53:22.368327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840766,"stat_sum":{"num_bytes":39037560,"num_objects":65,"num_object_clones":11,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":37,"num_read":5316,"num_read_kb":10680,"num_write":483,"num_write_kb":3561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"145'375","reported_seq":8503,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.002902+0000","last_change":"2024-05-06T00:28:31.628743+0000","last_active":"2024-05-06T00:29:27.002902+0000","last_peered":"2024-05-06T00:29:27.002902+0000","last_clean":"2024-05-06T00:29:27.002902+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:29:27.002902+0000","last_undegraded":"2024-05-06T00:29:27.002902+0000","last_fullsized":"2024-05-06T00:29:27.002902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:38:04.717150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094277900000000003,"stat_sum":{"num_bytes":54801920,"num_objects":81,"num_object_clones":20,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":45,"num_read":8415,"num_read_kb":21046,"num_write":333,"num_write_kb":3785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"145'436","reported_seq":8563,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-05-06T00:29:27.320376+0000","last_change":"2024-05-06T00:28:31.034254+0000","last_active":"2024-05-06T00:29:27.320376+0000","last_peered":"2024-05-06T00:29:27.320376+0000","last_clean":"2024-05-06T00:29:27.320376+0000","last_became_active":"2024-05-06T00:28:31.032323+0000","last_became_peered":"2024-05-06T00:28:31.032323+0000","last_unstale":"2024-05-06T00:29:27.320376+0000","last_undegraded":"2024-05-06T00:29:27.320376+0000","last_fullsized":"2024-05-06T00:29:27.320376+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:16:04.650752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019630709999999998,"stat_sum":{"num_bytes":40513121,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":23,"num_read":9448,"num_read_kb":38221,"num_write":353,"num_write_kb":5403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1305464219,"num_objects":2124,"num_object_clones":502,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":1030,"num_read":237742,"num_read_kb":786154,"num_write":16619,"num_write_kb":183136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5546,"num_bytes_recovered":71938434,"num_keys_recovered":904,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":136790016,"data_stored":171652918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15337,"ondisk_log_size":15337,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640297,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55568,"kb_used_data":24388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316272,"statfs":{"total":96636764160,"available":96579862528,"internally_reserved":0,"allocated":24973312,"data_stored":28385762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108888,"kb_used_data":29136,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94262952,"statfs":{"total":96636764160,"available":96525262848,"internally_reserved":0,"allocated":29835264,"data_stored":37194192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640295,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76976,"kb_used_data":23592,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94294864,"statfs":{"total":96636764160,"available":96557940736,"internally_reserved":0,"allocated":24158208,"data_stored":27555465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640296,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80664,"kb_used_data":23108,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94291176,"statfs":{"total":96636764160,"available":96554164224,"internally_reserved":0,"allocated":23662592,"data_stored":28543390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640297,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79488,"kb_used_data":21940,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94292352,"statfs":{"total":96636764160,"available":96555368448,"internally_reserved":0,"allocated":22466560,"data_stored":29004187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640295,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79592,"kb_used_data":26204,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94292248,"statfs":{"total":96636764160,"available":96555261952,"internally_reserved":0,"allocated":26832896,"data_stored":30044596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24399872,"data_stored":28630212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20766720,"data_stored":28312871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20144128,"data_stored":26049322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20357120,"data_stored":24754596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28180480,"data_stored":36539755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22941696,"data_stored":27366162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:29:31.658 INFO:tasks.thrashosds.thrasher:Setting 3.18 to [5, 1, 4, 2] 2024-05-06T00:29:31.658 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.18', '5', '1', '4', '2'] 2024-05-06T00:29:31.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.18 5 1 4 2 2024-05-06T00:29:31.667 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:31.743 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:31.742+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:31.825 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:31.825 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:31.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:31.839 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:31.844 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:31.842+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:31.882 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:29:31.945 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:31.946+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:32.003 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:32.003 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:32.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:32.064 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:32.065+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:32.135 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:32.136 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:32.147 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:32.145+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:32.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:32.182 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:32.182 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:32.199 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:32.199 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:32.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:32.216 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:32.236 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:32.237 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:32.248 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:32.246+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:32.260 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:32.283 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:32.284 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.30894476, 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:29:32.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:32.319 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.30871618800001, 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:29:32.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.30863890400001, 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.308442923, 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:29:32.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 91.401333546999993, 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.333 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:32.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:29:32.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.337 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 91.401272668000004, 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:29:32.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:32.342 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.283716217999995, 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:29:32.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:32.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.283671490000003, 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:29:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:29:32.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:29:32.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.283633089000006, 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.283595826999999, 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:32.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.283558917999997, 2024-05-06T00:29:32.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:29:32.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:29:32.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.283520999999993, 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:29:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:29:32.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.283483884999995, 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.369 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:29:32.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:29:32.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.277703435999996, 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:29:32.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.277665352, 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:29:32.378 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.382 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.277627678000002, 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:29:32.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.268192167999999, 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:32.387 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:29:32.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.268159741000005, 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:29:32.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.268135858999997, 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:32.397 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:32.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.268115929000004, 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:32.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:32.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:32.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:32.405 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:32.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:32.408 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:32.405+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:32.508 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:32.506+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:32.538 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:32.538 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:32.538 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:32.538 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:32.551 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:32.641 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:32.642+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:32.710 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:32.709+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:32.716 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:32.716 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:32.716 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:32.716 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:32.716 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:32.721 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #41, image image_client.0-clone41 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent42 2024-05-06T00:29:32.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:32.821 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:32.822+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:set 3.18 pg_upmap_items mapping to [5->1,4->2] 2024-05-06T00:29:32.911 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:32.910+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 183.31165955099999, 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:32.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.922 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 135.03897567600001, 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:29:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 135.03888699699999, 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:29:32.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:29:32.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 135.03878505399999, 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:32.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:29:32.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:29:32.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 135.03867595200001, 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:29:32.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 135.03857809300001, 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.947 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:29:32.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 135.03851304299999, 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:29:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 135.03842039700001, 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.959 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:29:32.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:29:32.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:29:32.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 135.03834839300001, 2024-05-06T00:29:32.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:32.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.27888690699999, 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.971 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:29:32.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:29:32.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.27886765900001, 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.977 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.27881753899999, 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:29:32.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.988 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.27879648800001, 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:32.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:29:32.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:29:32.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:32.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.27878116900001, 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:29:32.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:29:32.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:29:32.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.278764287, 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:33.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:29:33.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:29:33.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.27874595099999, 2024-05-06T00:29:33.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:29:33.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:29:33.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:29:33.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:29:33.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.278729949, 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:33.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:29:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:29:33.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.27864358599999, 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:29:33.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:29:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [read 12582912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.099393340000001, 2024-05-06T00:29:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857773900000002, 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.812763+0000", 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99789039000000002 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813015+0000", 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025184099999999999 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813243+0000", 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022833400000000001 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [sparse-read 12587008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.099292867999999, 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99922042799999999, 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:29:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813844+0000", 2024-05-06T00:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99888496199999999 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813901+0000", 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6561000000000003e-05 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813986+0000", 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5643999999999995e-05 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.035 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.035 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:33.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:33.037 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:33.037+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:33.137 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:33.137+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:33.225 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:33.225 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:33.225 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:33.226 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:33.238 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:33.238+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:33.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:33.339 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:33.337+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:33.409 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:33.409 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:33.410 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:33.410 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:33.410 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:33.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:33.440 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:33.437+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:33.540 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:33.541+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.890775885, 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.553 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:33.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.557 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.911534380000006, 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.558 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.911460160999994, 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.562 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.911398262000006, 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.911338662999995, 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.570 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:33.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:33.574 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.911254528000001, 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.575 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:33.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:33.579 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.911189015000005, 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:33.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:33.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.911131152999999, 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.584 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:33.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.911034557999997, 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:33.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910956833, 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.593 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:33.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:33.597 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910884046999996, 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:33.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:33.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910806555999997, 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910712426999993, 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.606 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:33.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910629465, 2024-05-06T00:29:33.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:33.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:33.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910555360000004, 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910456171000007, 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.624 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910393018999997, 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:33.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910292753999997, 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.629 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:33.633 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.910203382000006, 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:33.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.664344698999997, 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.638 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:33.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:33.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:33.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:33.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:33.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:33.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:33.642 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:33.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:33.644 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:33.641+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:33.745 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:33.745+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:33.778 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:33.779 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:33.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:33.847 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:33.845+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:33.914 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:29:33.918 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:33.918 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:33.918 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:33.918 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:33.918 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:33.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:33.975 INFO:teuthology.orchestra.run.smithi186.stdout:5899 punch from 0x2655000 to 0x2664c00, (0xfc00 bytes) 2024-05-06T00:29:33.977 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:33.978+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:33.994 INFO:teuthology.orchestra.run.smithi186.stdout:5900 trunc from 0xdcb4c00 to 0xedcf000 2024-05-06T00:29:34.008 INFO:teuthology.orchestra.run.smithi186.stdout:5902 write 0x7103000 thru 0x7109c3a (0x6c3b bytes) 2024-05-06T00:29:34.009 INFO:teuthology.orchestra.run.smithi186.stdout:5903 write 0x3585e00 thru 0x358bb20 (0x5d21 bytes) 2024-05-06T00:29:34.010 INFO:teuthology.orchestra.run.smithi186.stdout:5904 punch from 0x7d6d400 to 0x7d73e00, (0x6a00 bytes) 2024-05-06T00:29:34.046 INFO:teuthology.orchestra.run.smithi186.stdout:5906 trunc from 0xedcf000 to 0x7995200 2024-05-06T00:29:34.048 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:34.046+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:34.059 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:34.059 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:34.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:34.059 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.097904811999996, 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.060 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:34.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.096720196999996, 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.065 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:34.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:34.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.213299152000001, 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:29:34.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.074 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.213261938999999, 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.075 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.213173133000002, 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.079 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.213112940999999, 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.213058638, 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.088 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:29:34.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.212991998, 2024-05-06T00:29:34.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:29:34.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.096 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.212929725999999, 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:29:34.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.212870729999999, 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.101 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.105 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.206466009, 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:29:34.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.206417275, 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:29:34.110 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:29:34.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.206373023000001, 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:29:34.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:29:34.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.206301991, 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:29:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.206251175999999, 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.123 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.197512027999998, 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.197489013999999, 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:29:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:29:34.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.197466764000001, 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.136 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:29:34.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:29:34.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.197356914, 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:29:34.140 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:29:34.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:29:34.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.197292864000001, 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.148 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:34.149 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:34.149 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:34.151 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:34.150+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:34.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:34.152 INFO:teuthology.orchestra.run.smithi186.stdout:5908 read 0x4df4600 thru 0x4e02047 (0xda48 bytes) 2024-05-06T00:29:34.152 INFO:teuthology.orchestra.run.smithi186.stdout:5910 read 0x5488e00 thru 0x5492768 (0x9969 bytes) 2024-05-06T00:29:34.152 INFO:teuthology.orchestra.run.smithi186.stdout:5911 write 0x59a5400 thru 0x59b446f (0xf070 bytes) 2024-05-06T00:29:34.152 INFO:teuthology.orchestra.run.smithi186.stdout:5912 read 0x21fde00 thru 0x2201340 (0x3541 bytes) 2024-05-06T00:29:34.152 INFO:teuthology.orchestra.run.smithi186.stdout:5919 write 0xcf13e00 thru 0xcf23656 (0xf857 bytes) 2024-05-06T00:29:34.152 INFO:teuthology.orchestra.run.smithi186.stdout:5924 write 0xac06200 thru 0xac15651 (0xf452 bytes) 2024-05-06T00:29:34.152 INFO:teuthology.orchestra.run.smithi186.stdout:5925 read 0x225800 thru 0x23385d (0xe05e bytes) 2024-05-06T00:29:34.152 INFO:teuthology.orchestra.run.smithi186.stdout:5928 trunc from 0xcf23800 to 0x9aab800 2024-05-06T00:29:34.177 INFO:teuthology.orchestra.run.smithi186.stdout:5929 write 0xc5c9a00 thru 0xc5d5169 (0xb76a bytes) 2024-05-06T00:29:34.179 INFO:teuthology.orchestra.run.smithi186.stdout:5930 write 0x6cb6c00 thru 0x6cc1400 (0xa801 bytes) 2024-05-06T00:29:34.179 INFO:teuthology.orchestra.run.smithi186.stdout:5932 read 0xa727e00 thru 0xa733f3f (0xc140 bytes) 2024-05-06T00:29:34.179 INFO:teuthology.orchestra.run.smithi186.stdout:5934 read 0xbb28000 thru 0xbb2e3f2 (0x63f3 bytes) 2024-05-06T00:29:34.180 INFO:teuthology.orchestra.run.smithi186.stdout:5935 read 0xc1bfe00 thru 0xc1cda75 (0xdc76 bytes) 2024-05-06T00:29:34.180 INFO:teuthology.orchestra.run.smithi186.stdout:5937 write 0x93400 thru 0x96c58 (0x3859 bytes) 2024-05-06T00:29:34.181 INFO:teuthology.orchestra.run.smithi186.stdout:5939 write 0x4827400 thru 0x482a060 (0x2c61 bytes) 2024-05-06T00:29:34.199 INFO:teuthology.orchestra.run.smithi186.stdout:5941 trunc from 0xc5d5200 to 0x61dae00 2024-05-06T00:29:34.251 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:34.249+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:34.279 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:34.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:34.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:34.290 INFO:teuthology.orchestra.run.smithi186.stdout:5943 write 0xa737000 thru 0xa73c157 (0x5158 bytes) 2024-05-06T00:29:34.294 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:34.303 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:34.346 INFO:teuthology.orchestra.run.smithi186.stdout:5946 trunc from 0xa73c200 to 0x2818c00 2024-05-06T00:29:34.352 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:34.350+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:34.391 INFO:teuthology.orchestra.run.smithi186.stdout:5947 read 0x10bbc00 thru 0x10bf1c0 (0x35c1 bytes) 2024-05-06T00:29:34.402 INFO:teuthology.orchestra.run.smithi186.stdout:5948 trunc from 0x2818c00 to 0x63d8000 2024-05-06T00:29:34.430 INFO:teuthology.orchestra.run.smithi186.stdout:5950 trunc from 0x63d8000 to 0x2645000 2024-05-06T00:29:34.452 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:34.453+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:34.466 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:34.467 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:34.474 INFO:teuthology.orchestra.run.smithi186.stdout:5952 punch from 0xaa00 to 0x19a00, (0xf000 bytes) 2024-05-06T00:29:34.488 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:34.490 INFO:teuthology.orchestra.run.smithi186.stdout:5953 trunc from 0x2645000 to 0x771c600 2024-05-06T00:29:34.497 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:34.498 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:34.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:34.517 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:34.523 INFO:teuthology.orchestra.run.smithi186.stdout:5955 trunc from 0x771c600 to 0xbcb7600 2024-05-06T00:29:34.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:34.536 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:34.537 INFO:teuthology.orchestra.run.smithi186.stdout:5958 write 0x712e200 thru 0x7138784 (0xa585 bytes) 2024-05-06T00:29:34.543 INFO:teuthology.orchestra.run.smithi186.stdout:5959 trunc from 0xbcb7600 to 0xe040600 2024-05-06T00:29:34.553 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:34.554+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:34.557 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:34.558 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:34.568 INFO:teuthology.orchestra.run.smithi186.stdout:5960 punch from 0x233c600 to 0x234a600, (0xe000 bytes) 2024-05-06T00:29:34.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:34.583 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:34.586 INFO:teuthology.orchestra.run.smithi186.stdout:5961 trunc from 0xe040600 to 0x5a24e00 2024-05-06T00:29:34.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:34.606 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:34.615 INFO:teuthology.orchestra.run.smithi186.stdout:5963 read 0x329ca00 thru 0x32ab555 (0xeb56 bytes) 2024-05-06T00:29:34.617 INFO:teuthology.orchestra.run.smithi186.stdout:5964 read 0xfb000 thru 0xff3d1 (0x43d2 bytes) 2024-05-06T00:29:34.621 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:34.621 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:34.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:34.642 INFO:teuthology.orchestra.run.smithi186.stdout:5965 write 0x98f1000 thru 0x98fe2d0 (0xd2d1 bytes) 2024-05-06T00:29:34.644 INFO:teuthology.orchestra.run.smithi186.stdout:5968 punch from 0x71a2c00 to 0x71a6600, (0x3a00 bytes) 2024-05-06T00:29:34.646 INFO:teuthology.orchestra.run.smithi186.stdout:5970 punch from 0x54f1e00 to 0x54f3000, (0x1200 bytes) 2024-05-06T00:29:34.646 INFO:teuthology.orchestra.run.smithi186.stdout:5971 punch from 0x39d600 to 0x3a6e00, (0x9800 bytes) 2024-05-06T00:29:34.667 INFO:teuthology.orchestra.run.smithi186.stdout:5973 trunc from 0x98fe400 to 0x13c5600 2024-05-06T00:29:34.680 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:34.681+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:34.710 INFO:teuthology.orchestra.run.smithi186.stdout:5974 punch from 0x468800 to 0x46b000, (0x2800 bytes) 2024-05-06T00:29:34.712 INFO:teuthology.orchestra.run.smithi186.stdout:5975 punch from 0xf5f800 to 0xf62e00, (0x3600 bytes) 2024-05-06T00:29:34.712 INFO:teuthology.orchestra.run.smithi186.stdout:5976 read 0xcdb800 thru 0xce0963 (0x5164 bytes) 2024-05-06T00:29:34.714 INFO:teuthology.orchestra.run.smithi186.stdout:5977 punch from 0xfe0800 to 0xfe8800, (0x8000 bytes) 2024-05-06T00:29:34.743 INFO:teuthology.orchestra.run.smithi186.stdout:5978 write 0x4a77200 thru 0x4a86e6c (0xfc6d bytes) 2024-05-06T00:29:34.745 INFO:teuthology.orchestra.run.smithi186.stdout:5981 write 0x5d2a00 thru 0x5e1b12 (0xf113 bytes) 2024-05-06T00:29:34.746 INFO:teuthology.orchestra.run.smithi186.stdout:5983 read 0x3e86600 thru 0x3e871f1 (0xbf2 bytes) 2024-05-06T00:29:34.755 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:34.754+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:34.766 INFO:teuthology.orchestra.run.smithi186.stdout:5984 trunc from 0x4a87000 to 0xcafee00 2024-05-06T00:29:34.798 INFO:teuthology.orchestra.run.smithi186.stdout:5985 write 0x2dd2a00 thru 0x2dd6071 (0x3672 bytes) 2024-05-06T00:29:34.801 INFO:teuthology.orchestra.run.smithi186.stdout:5988 read 0x7d82400 thru 0x7d829ae (0x5af bytes) 2024-05-06T00:29:34.801 INFO:teuthology.orchestra.run.smithi186.stdout:5991 write 0x9e11a00 thru 0x9e11c6f (0x270 bytes) 2024-05-06T00:29:34.801 INFO:teuthology.orchestra.run.smithi186.stdout:5993 write 0x90f4800 thru 0x90f5e72 (0x1673 bytes) 2024-05-06T00:29:34.801 INFO:teuthology.orchestra.run.smithi186.stdout:5994 read 0xcf5a00 thru 0xd025c3 (0xcbc4 bytes) 2024-05-06T00:29:34.806 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:34.806 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:34.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:34.821 INFO:teuthology.orchestra.run.smithi186.stdout:5995 trunc from 0xcafee00 to 0x18fd800 2024-05-06T00:29:34.857 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:34.854+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:34.860 INFO:teuthology.orchestra.run.smithi186.stdout:5997 punch from 0xeb1800 to 0xeb9400, (0x7c00 bytes) 2024-05-06T00:29:34.865 INFO:teuthology.orchestra.run.smithi186.stdout:5998 trunc from 0x18fd800 to 0x2e1b000 2024-05-06T00:29:34.882 INFO:teuthology.orchestra.run.smithi186.stdout:5999 read 0x1342800 thru 0x13482d9 (0x5ada bytes) 2024-05-06T00:29:34.886 INFO:teuthology.orchestra.run.smithi186.stdout:6000 read 0x28dfc00 thru 0x28e1f12 (0x2313 bytes) 2024-05-06T00:29:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:29:34.948 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:34.948 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:34.957 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:34.957+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:34.963 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:35.058 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:35.058+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:35.114 INFO:teuthology.orchestra.run.smithi186.stdout:checking clone #42, image image_client.0-clone42 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent43 2024-05-06T00:29:35.128 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:35.128 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:35.143 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:35.188 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:35.189+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:35.262 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:35.261+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:35.264 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:35.265 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:35.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:35.368 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:35.369+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:35.455 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:35.455 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:35.465 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:35.466+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:35.472 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:35.566 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:35.565+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:35.658 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:35.658 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:35.669 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:35.665+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:35.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:35.768 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:35.766+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:35.811 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:35.811 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:35.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:35.869 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:35.869+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:35.970 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:35.969+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:35.996 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:35.997 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:36.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:36.071 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:36.070+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:36.140 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:36.141 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:36.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:36.197 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:36.198+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:36.273 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:36.273+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:36.324 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:36.324 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:36.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:36.389 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:36.390+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:36.461 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:36.461 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:36.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:36.476 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:36.473+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:36.576 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:36.577+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:36.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:36.634 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:36.634 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:36.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:36.698 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:36.698+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:36.775 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:36.775 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:36.777 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:36.778+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:36.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:36.878 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:36.877+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:36.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:29:36.958 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:36.959 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:36.975 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:36.979 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:36.977+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:36.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:36.990 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:37.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:37.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:37.047 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:37.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:37.071 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:37.079 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:37.078+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:37.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:37.091 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:37.103 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:37.103 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:37.110 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:37.110 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:37.117 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:37.180 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:37.181+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:37.234 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:37.234 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:37.235 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:37.235 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:37.249 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:37.292 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:37.293+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:37.383 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:37.381+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:37.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:37.409 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:37.410 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:37.410 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:37.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:37.508 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:37.509+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:37.582 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 158.778259531, 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:29:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:29:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:29:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:29:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 158.778243328, 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:29:37.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.770013110999997, 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:29:37.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:29:37.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769981740999995, 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:29:37.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:37.605 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769947951999995, 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:37.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:29:37.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:29:37.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769767496, 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:29:37.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769733853000005, 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769710071000006, 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:29:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769683794000002, 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:29:37.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769640553000002, 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:29:37.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:29:37.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:37.636 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769610924000006, 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:37.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:29:37.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769578073999995, 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:37.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.646 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769546435999999, 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:29:37.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.652 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769513634000006, 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:29:37.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.657 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769481600999995, 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:29:37.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769449522000002, 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:29:37.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.769418213999998, 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:37.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:29:37.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:37.672 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.929272496999999, 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.927866348000002, 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:29:37.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:29:37.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:37.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.927813389999997, 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.685 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:29:37.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:37.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:37.690 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:37.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:37.693 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:37.693+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:37.794 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:37.794+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:37.863 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:37.863 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:37.863 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:37.863 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:37.877 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:37.895 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:37.894+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:37.904 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:29:37.911 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:29:37.911 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:29:37.911 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-05-06T00:29:37.912 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-06T00:29:37.912 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-06T00:29:37.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-05-06T00:29:37.995 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:37.993+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:38.017 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:38.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:38.017 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:38.017 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:38.018 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:38.032 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:38.035 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-06T00:29:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:38.096 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:38.097+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:38.169 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:38.169 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:38.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:38.169 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:38.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.169 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.141177599000002, 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:38.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:29:38.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.141156809999998, 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:38.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:29:38.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:29:38.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.141136747999994, 2024-05-06T00:29:38.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:29:38.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.141117472000005, 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:38.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:29:38.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:29:38.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.141097353999996, 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:29:38.187 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:29:38.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.141065944999994, 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.141033032999999, 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:29:38.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.141012446000005, 2024-05-06T00:29:38.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:29:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.140993065999993, 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:29:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:29:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.140948385000002, 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.210 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:29:38.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.140883294000005, 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.215 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:29:38.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.219 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.140851720000001, 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:38.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:29:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:29:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:29:38.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.140829459000003, 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.140808343000003, 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.229 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 93.483091716999994, 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:29:38.233 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:29:38.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 93.482681475000007, 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.238 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.544036880999997, 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.243 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:29:38.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:29:38.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.543965597000003, 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:29:38.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:29:38.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.543933963999997, 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:38.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:29:38.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.539715418, 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:38.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:29:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:29:38.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:29:38.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.265 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.265 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:38.266 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:38.267 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:38.265+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:38.367 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:38.365+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:38.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:38.436 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:38.450 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:38.468 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:38.466+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:38.569 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:38.569+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:38.617 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:38.617 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:38.617 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:38.618 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:38.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:38.674 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:38.673+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:38.770 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:38.770+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:38.793 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.78597663799999, 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:29:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.785748066, 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:29:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:29:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.78567078200001, 2024-05-06T00:29:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:29:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:29:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.78547480099999, 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:29:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.878365424999998, 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:29:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:29:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.878304545999995, 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:29:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:29:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.760748096, 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.760703367999994, 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:29:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.760664966999997, 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:29:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:29:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:29:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:29:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:29:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.760627705000005, 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:29:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.760590796000002, 2024-05-06T00:29:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:29:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:29:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.760552877999999, 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:29:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:29:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.760515763000001, 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:29:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.754735314000001, 2024-05-06T00:29:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:29:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.754697230000005, 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:29:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:29:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.754659556000007, 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.745224046000004, 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:29:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.745191618999996, 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.745167737000003, 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:29:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:29:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 94.745147806999995, 2024-05-06T00:29:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:38.887 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:38.888 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:38.889 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:38.889+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:38.989 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:38.990+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:39.029 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:39.030 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:39.030 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:39.030 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:39.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:39.089 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:39.090+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:39.129 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:39.190 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:39.190+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:39.218 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:39.219 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:39.219 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:39.219 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:39.219 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:39.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:39.287 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:39.287 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:39.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:39.304 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:39.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:39.320 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:39.321 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:39.322+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:39.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:39.341 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:39.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:39.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:39.382 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:39.392 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:39.390+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:39.401 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:39.402 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:39.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:39.493 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:39.493+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:39.579 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:39.579 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:39.594 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:39.593+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:39.595 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:39.695 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:39.693+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:39.718 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:39.718 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:39.732 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:39.796 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:39.794+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:39.860 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:39.860 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:39.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:39.897 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:39.897+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:39.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:29:39.999 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:39.998+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:40.045 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:40.045 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:40.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:40.100 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:40.101+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:40.201 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:40.201+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:40.219 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:40.219 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:40.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:40.302 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:40.302+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:40.352 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:40.352 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:40.366 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:40.413 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:40.414+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:40.492 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:40.492 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:40.504 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:40.505+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:40.506 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:40.605 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:40.606+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:40.672 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:40.672 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:40.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:40.732 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:40.733+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:40.808 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:40.808 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:40.809 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:40.806+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:40.821 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:40.909 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:40.910+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:40.964 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:29:41.004 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:41.005 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:41.009 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:41.010+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:41.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:41.110 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:41.109+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:41.187 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:41.187 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:41.203 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:41.248 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:41.249+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:41.313 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:41.314+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:41.368 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:41.368 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:41.382 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:41.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:41.414 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:41.414+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:41.505 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:41.505 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:41.515 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:41.513+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:41.520 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:41.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:41.564 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:41.580 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:41.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:41.607 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:41.615 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:41.614+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:41.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:41.632 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:41.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:41.656 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:41.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:41.677 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:41.688 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:41.688 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:41.688 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:41.688 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:41.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:41.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:41.717 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:41.718+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:41.817 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:41.818+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:41.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.867816573, 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:29:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.867588001, 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:29:41.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:29:41.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:41.884 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.86751071699999, 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:41.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:41.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:29:41.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:29:41.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:41.891 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.867314736, 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:29:41.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 100.96020536, 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:29:41.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 100.960144481, 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:29:41.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:41.905 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.842588031000005, 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:41.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:29:41.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.842543302999999, 2024-05-06T00:29:41.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:41.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:29:41.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.842504902000002, 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.842467639999995, 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:29:41.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.842430730999993, 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:29:41.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:29:41.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.842392813000004, 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.842355698000006, 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:29:41.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.836575249000006, 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.836537164999996, 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:29:41.941 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:41.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:29:41.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.836499490999998, 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:29:41.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:29:41.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.949 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.827063980999995, 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:41.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:29:41.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:29:41.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.827031554000001, 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.827007671999993, 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:29:41.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.826987742, 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:41.961 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:41.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:29:41.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:41.965 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:41.965 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:41.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:41.967 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:41.966+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:42.068 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:42.066+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:42.098 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:42.098 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:42.098 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:42.098 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:42.111 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:42.168 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:42.169+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:42.269 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:42.269+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:42.275 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:42.275 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:42.275 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:42.275 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:42.275 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:42.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:42.371 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:42.370+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:42.416 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:42.416 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:42.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:42.416 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:42.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.416 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:29:42.416 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 192.811301122, 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:29:42.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:29:42.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:29:42.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 144.53861724699999, 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.424 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:29:42.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:29:42.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:29:42.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.430 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:42.430 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:42.430 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 144.538528568, 2024-05-06T00:29:42.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:29:42.430 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.430 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.430 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.430 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:29:42.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:29:42.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 144.538426625, 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:29:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 144.53831752299999, 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:29:42.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.447 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 144.538219664, 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:29:42.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:29:42.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:29:42.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 144.53815461400001, 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:29:42.453 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:29:42.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 144.53806196799999, 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.459 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:42.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:29:42.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 144.53798996399999, 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.778528478, 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.471 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:42.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:29:42.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.476 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.77850923, 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:29:42.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:29:42.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:29:42.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.482 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.77845911, 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:29:42.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:29:42.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.778438059, 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:29:42.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.494 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.77842274, 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.500 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.77840585800001, 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.778387522, 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.506 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:42.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:29:42.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:29:42.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:29:42.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.77837152000001, 2024-05-06T00:29:42.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 143.778285157, 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.518 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:29:42.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:42.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [read 12582912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.599034911, 2024-05-06T00:29:42.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857773900000002, 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:42.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.812763+0000", 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99789039000000002 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813015+0000", 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025184099999999999 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813243+0000", 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022833400000000001 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [sparse-read 12587008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.598934439000001, 2024-05-06T00:29:42.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99922042799999999, 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:29:42.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813844+0000", 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99888496199999999 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813901+0000", 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6561000000000003e-05 2024-05-06T00:29:42.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813986+0000", 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5643999999999995e-05 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:42.534 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:42.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:42.536 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:42.533+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:42.638 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:42.637+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:42.715 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:42.715 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:42.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:42.777 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:42.777+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:42.841 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:42.841+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:42.858 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:42.858 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:42.858 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:42.859 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:42.859 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:42.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:42.942 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:42.941+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:42.990 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:29:43.039 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:43.039 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:43.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.377661338, 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:43.040 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:43.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:43.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:43.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.046 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.047 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:43.047 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.39841983300001, 2024-05-06T00:29:43.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:43.048 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.048 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.048 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.049 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:43.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.049 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:43.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:43.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:43.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:43.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:43.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:43.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:43.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:43.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:43.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:43.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:43.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:43.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:43.057 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.057 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.058 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.058 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:43.058 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.39834561399999, 2024-05-06T00:29:43.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:43.059 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.059 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.059 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.059 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.059 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.060 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:43.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.060 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:43.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:43.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:43.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:43.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:43.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:43.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:43.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:43.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:43.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:43.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.066 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:43.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.39828371500001, 2024-05-06T00:29:43.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:43.067 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:43.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.068 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:43.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.39822411599999, 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.073 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:43.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:43.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:43.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:43.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:43.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:43.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:43.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:43.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.080 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.081 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.081 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:43.081 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.398139981, 2024-05-06T00:29:43.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:43.081 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.081 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.082 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.082 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:43.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.083 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:43.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:43.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:43.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:43.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:43.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:43.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:43.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:43.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:43.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:43.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.088 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.398074468, 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.398016606, 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.093 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:43.097 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.397920011, 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:43.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:43.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:43.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.397842286, 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.3977695, 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.106 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.397692009, 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:43.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.39759788000001, 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.397514918, 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.123 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.397440813, 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:43.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:43.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.39734162400001, 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:43.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:43.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.397278472, 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.397178207, 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:43.140 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.39708883500001, 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:43.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.151230152, 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:43.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:43.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:43.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.149 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.149 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.149 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:43.149 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:29:43.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-05-06T00:29:43.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:43.151 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:43.149+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:43.251 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:43.249+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:43.322 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:43.322 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:43.322 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:43.322 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:43.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:43.352 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:43.353+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:43.454 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:43.453+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:43.457 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:43.457 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:43.457 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:43.457 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:43.457 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:43.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:43.555 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:43.553+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:43.628 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.1 to 1 (10000) 2024-05-06T00:29:43.634 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:29:43.638 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:43.638 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:43.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.677036096999998, 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:43.639 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:43.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:43.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:43.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:43.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:43.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:43.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:43.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:43.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:43.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:43.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:43.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:43.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:43.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:43.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.675851481999999, 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:43.648 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:43.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:43.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:43.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:43.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:43.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:43.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:43.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:43.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:43.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:43.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:43.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:43.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:43.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:43.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.655 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.657 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.657 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:43.658 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.792430437, 2024-05-06T00:29:43.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:29:43.658 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.658 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.658 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:43.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:29:43.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:29:43.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:29:43.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:29:43.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:29:43.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:29:43.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.664 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.792393224000001, 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.669 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.792304418000001, 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:29:43.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:29:43.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.792244226000001, 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.674 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:29:43.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:29:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.792189922999999, 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.679 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:29:43.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.792123282999999, 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.684 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:29:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:29:43.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.792061011000001, 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.688 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:29:43.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.792002015000001, 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:29:43.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.785597293999999, 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:29:43.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.699 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:43.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:29:43.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.785548559999999, 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.703 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:29:43.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.707 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.785504308, 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:29:43.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:29:43.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.785433275999999, 2024-05-06T00:29:43.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:29:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.785382461000001, 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:29:43.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:29:43.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.776643313000001, 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:29:43.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:43.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:29:43.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:29:43.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.776620299000001, 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:29:43.727 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:29:43.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:29:43.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:29:43.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.776598049, 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:29:43.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.776488198999999, 2024-05-06T00:29:43.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:43.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:29:43.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:29:43.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:29:43.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:29:43.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:29:43.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.741 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.742 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:43.742 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:43.742 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.776424149, 2024-05-06T00:29:43.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:29:43.742 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:43.743 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:43.743 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:43.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:43.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:43.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:43.743 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:29:43.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.744 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:43.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:43.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:43.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:43.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:43.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:43.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:43.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:29:43.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:29:43.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:43.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:29:43.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:29:43.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:43.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:29:43.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:29:43.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:29:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:29:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:43.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:29:43.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:29:43.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:43.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:29:43.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:43.750 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:43.751 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:43.753 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-05-06T00:29:43.754 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:43.749+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:43.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:43.853 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:43.853+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:43.929 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:43.929 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:43.946 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:43.947 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:43.947 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:43.953 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:43.953+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:43.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:43.966 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:43.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:43.984 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:44.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:44.006 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:44.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:44.027 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:44.048 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:44.054 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:44.053+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:44.063 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:44.063 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:44.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:44.168 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:44.169+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:44.239 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:44.240 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:44.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:44.257 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:44.257+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:44.358 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:44.357+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:44.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:44.459 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:44.457+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:44.560 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:44.557+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:44.598 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:44.598 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:44.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.4 to 1 (10000) 2024-05-06T00:29:44.656 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-05-06T00:29:44.660 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:44.661+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:44.738 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:44.738 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:44.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:44.762 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:44.761+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:44.863 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:44.861+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:44.920 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:44.920 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:44.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:44.980 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:44.981+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:45.049 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:45.049 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:45.063 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:45.104 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:45.105+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:45.166 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:45.165+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:45.225 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:45.225 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:45.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:45.266 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:45.265+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:45.367 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:45.369+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:45.369 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:45.370 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:45.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:45.469 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:45.469+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:45.505 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:45.506 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:45.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:45.570 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:45.569+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:45.639 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.0 to 1 (10000) 2024-05-06T00:29:45.648 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:45.648 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:45.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-05-06T00:29:45.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:45.712 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:45.713+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:45.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:29:45.804 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:45.805+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:45.842 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:45.842 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:45.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:45.872 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:45.873+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:45.973 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:45.973+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:45.978 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:45.978 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:45.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:46.067 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:46.080 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:46.081+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:46.151 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:46.152 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:46.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:46.197 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:46.209 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:46.209+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:46.213 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:46.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:46.264 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:46.265 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:46.276 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:46.273+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:46.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:46.282 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:46.333 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:46.333 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:46.333 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:46.333 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:46.347 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:46.389 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:46.389+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:46.478 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:46.477+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:46.513 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:46.513 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:46.513 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:46.513 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:46.513 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:46.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:46.617 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:46.617+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:46.628 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.5 to 1 (10000) 2024-05-06T00:29:46.628 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:29:46.662 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-05-06T00:29:46.679 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:46.677+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:46.702 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.740058579000006, 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:46.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:46.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.708 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.738873964000007, 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:46.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.855452919000001, 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:29:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:29:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:29:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.855415706000002, 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:29:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:29:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.855326900000001, 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:29:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:29:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.855266708000002, 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:29:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:29:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.855212405000003, 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:29:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.855145765000003, 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.855083493000002, 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.855024497000002, 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:29:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.848619776, 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:29:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:29:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.848571042000003, 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:29:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:29:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.848526790000001, 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.848455758, 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.848404942999998, 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:29:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:29:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.839665795000002, 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:29:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.839642781000002, 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:29:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:29:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.839620531000001, 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:29:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.839510681, 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:29:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:29:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:29:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.839446631000001, 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:29:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:29:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:29:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:46.792 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:46.793 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:46.793+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:46.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:46.894 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:46.892+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:46.911 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:46.911 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:46.911 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:46.911 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:46.926 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:47.012 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:47.012+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:47.094 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:47.094 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:47.094 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:47.095 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:47.095 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:47.096 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:47.093+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:47.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:47.197 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:47.197+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:47.283 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:47.283 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:47.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:47.283 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:47.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 168.479433933, 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:29:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 168.47941772999999, 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:29:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.471187513, 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.471156143, 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:29:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:29:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:29:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:29:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.471122354, 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:29:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.47094189800001, 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:29:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.470908255, 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:29:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.470884473, 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:29:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:29:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.47085819599999, 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:29:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:29:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.47081495499999, 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:29:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:29:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:29:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.470785326, 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:29:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.470752476, 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:29:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.47072083800001, 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.470688036, 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:29:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:29:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.470656003, 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:29:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.47062392399999, 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:29:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.470592616, 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:29:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.630446898999999, 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:29:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.629040750000001, 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:29:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:29:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:29:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.628987791999997, 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:29:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.391 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:47.392 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:47.393 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:47.392+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:47.493 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:47.492+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:47.566 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:47.566 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:47.566 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:47.566 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:47.579 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:47.594 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:47.593+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.3 to 1 (10000) 2024-05-06T00:29:47.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-05-06T00:29:47.720 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:47.720+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:47.757 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:47.757 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:47.758 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:47.758 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:47.758 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:47.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:47.796 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:47.793+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:47.896 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:47.897+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.91124222000001, 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.911221431, 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:29:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:29:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.911201369, 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:29:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:29:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:29:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.91118209299999, 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:29:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:29:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.911161975, 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:29:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:29:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:29:47.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:29:47.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:29:47.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.911130566, 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:29:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.911097654, 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:29:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:29:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.91107706699999, 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:29:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.911057687, 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:29:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.911013006, 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:29:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:29:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:29:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:29:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.91094791499999, 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:29:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:29:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:29:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:29:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:29:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:29:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:29:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.910916341, 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:29:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:29:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.91089408000001, 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.91087296400001, 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.253156338, 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.252746096, 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:29:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.314101502, 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:29:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:29:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.314030217999999, 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.313998585, 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:29:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:29:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:29:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:29:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:29:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:29:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:29:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:29:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:29:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:29:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:29:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:29:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:29:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:29:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:29:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:29:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.309780039000003, 2024-05-06T00:29:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:29:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:29:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:48.043 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:48.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:48.046 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:48.045+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:48.146 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:48.145+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:48.161 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:48.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:48.161 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:48.161 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:48.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:48.246 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:48.244+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:48.294 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:48.295 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:48.295 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:48.295 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:48.295 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:48.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:48.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:48.352 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:48.352+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:48.429 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:48.445 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:48.447 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:48.448+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:48.461 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:48.469 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:48.469 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:48.482 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:48.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:48.483 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:48.502 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:48.519 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:48.520 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:48.578 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:48.576+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:48.615 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:48.615 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:48.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:48.650 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:48.648+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:48.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:29:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.2 to 1 (10000) 2024-05-06T00:29:48.708 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-05-06T00:29:48.708 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:29:48.751 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:48.749+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:48.756 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:48.757 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:48.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:48.865 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:48.865+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:48.942 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:48.943 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:48.953 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:48.952+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:48.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:49.036 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:29:49.037 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:29:49.053 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-05-06T00:29:48.674901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1033784243,"num_objects":1752,"num_object_clones":374,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":918,"num_read":255900,"num_read_kb":876333,"num_write":18601,"num_write_kb":193848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5554,"num_bytes_recovered":74299906,"num_keys_recovered":904,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17392,"ondisk_log_size":17392,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":437668,"kb_used_data":90952,"kb_used_omap":9,"kb_used_meta":346614,"kb_avail":565793372,"statfs":{"total":579820584960,"available":579372412928,"internally_reserved":0,"allocated":93134848,"data_stored":113302918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":354933408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-156482204,"num_objects":-139,"num_object_clones":-42,"num_object_copies":-278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-139,"num_whiteouts":-32,"num_read":1335,"num_read_kb":1141,"num_write":239,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.042335"},"pg_stats":[{"pgid":"3.1f","version":"160'604","reported_seq":12600,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.636280+0000","last_change":"2024-05-06T00:29:46.636553+0000","last_active":"2024-05-06T00:29:46.636280+0000","last_peered":"2024-05-06T00:29:46.636280+0000","last_clean":"2024-05-06T00:29:46.636280+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:29:46.636280+0000","last_undegraded":"2024-05-06T00:29:46.636280+0000","last_fullsized":"2024-05-06T00:29:46.636280+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:56:54.364195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001396659,"stat_sum":{"num_bytes":26140807,"num_objects":52,"num_object_clones":9,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":27,"num_read":12188,"num_read_kb":39835,"num_write":483,"num_write_kb":2565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1e","version":"160'310","reported_seq":9643,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.629362+0000","last_change":"2024-05-06T00:29:46.630009+0000","last_active":"2024-05-06T00:29:46.629362+0000","last_peered":"2024-05-06T00:29:46.629362+0000","last_clean":"2024-05-06T00:29:46.629362+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:29:46.629362+0000","last_undegraded":"2024-05-06T00:29:46.629362+0000","last_fullsized":"2024-05-06T00:29:46.629362+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:22:34.074033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011943030000000001,"stat_sum":{"num_bytes":73594399,"num_objects":66,"num_object_clones":19,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":28,"num_read":8756,"num_read_kb":50017,"num_write":277,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1d","version":"160'418","reported_seq":3693,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.687680+0000","last_change":"2024-05-06T00:29:46.688167+0000","last_active":"2024-05-06T00:29:46.687680+0000","last_peered":"2024-05-06T00:29:46.687680+0000","last_clean":"2024-05-06T00:29:46.687680+0000","last_became_active":"2024-05-06T00:29:13.784483+0000","last_became_peered":"2024-05-06T00:29:13.784483+0000","last_unstale":"2024-05-06T00:29:46.687680+0000","last_undegraded":"2024-05-06T00:29:46.687680+0000","last_fullsized":"2024-05-06T00:29:46.687680+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:32:50.803098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013667620000000001,"stat_sum":{"num_bytes":14054436,"num_objects":50,"num_object_clones":10,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":33,"num_read":3547,"num_read_kb":12883,"num_write":385,"num_write_kb":3277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1c","version":"161'788","reported_seq":9052,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.717953+0000","last_change":"2024-05-06T00:29:46.629058+0000","last_active":"2024-05-06T00:29:46.717953+0000","last_peered":"2024-05-06T00:29:46.717953+0000","last_clean":"2024-05-06T00:29:46.717953+0000","last_became_active":"2024-05-06T00:29:07.735705+0000","last_became_peered":"2024-05-06T00:29:07.735705+0000","last_unstale":"2024-05-06T00:29:46.717953+0000","last_undegraded":"2024-05-06T00:29:46.717953+0000","last_fullsized":"2024-05-06T00:29:46.717953+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:11:45.659856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001649944,"stat_sum":{"num_bytes":39499876,"num_objects":56,"num_object_clones":15,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":25,"num_read":12158,"num_read_kb":33759,"num_write":691,"num_write_kb":5259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1b","version":"160'1029","reported_seq":9360,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.637244+0000","last_change":"2024-05-06T00:29:46.637604+0000","last_active":"2024-05-06T00:29:46.637244+0000","last_peered":"2024-05-06T00:29:46.637244+0000","last_clean":"2024-05-06T00:29:46.637244+0000","last_became_active":"2024-05-06T00:28:05.820651+0000","last_became_peered":"2024-05-06T00:28:05.820651+0000","last_unstale":"2024-05-06T00:29:46.637244+0000","last_undegraded":"2024-05-06T00:29:46.637244+0000","last_fullsized":"2024-05-06T00:29:46.637244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1029,"log_dups_size":0,"ondisk_log_size":1029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:02:45.348907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002148582,"stat_sum":{"num_bytes":60492909,"num_objects":86,"num_object_clones":25,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":42,"num_read":12921,"num_read_kb":32938,"num_write":1023,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1a","version":"160'344","reported_seq":6667,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.798638+0000","last_change":"2024-05-06T00:29:46.627906+0000","last_active":"2024-05-06T00:29:46.798638+0000","last_peered":"2024-05-06T00:29:46.798638+0000","last_clean":"2024-05-06T00:29:46.798638+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:29:46.798638+0000","last_undegraded":"2024-05-06T00:29:46.798638+0000","last_fullsized":"2024-05-06T00:29:46.798638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:40:03.987219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001498105,"stat_sum":{"num_bytes":35118761,"num_objects":49,"num_object_clones":7,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":24,"num_read":6828,"num_read_kb":25214,"num_write":368,"num_write_kb":13634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.19","version":"160'611","reported_seq":3431,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.732005+0000","last_change":"2024-05-06T00:29:46.732277+0000","last_active":"2024-05-06T00:29:46.732005+0000","last_peered":"2024-05-06T00:29:46.732005+0000","last_clean":"2024-05-06T00:29:46.732005+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:29:46.732005+0000","last_undegraded":"2024-05-06T00:29:46.732005+0000","last_fullsized":"2024-05-06T00:29:46.732005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:40:27.203337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935834,"stat_sum":{"num_bytes":18421996,"num_objects":49,"num_object_clones":11,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":22,"num_read":4909,"num_read_kb":16684,"num_write":777,"num_write_kb":5645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.18","version":"161'656","reported_seq":14240,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.771675+0000","last_change":"2024-05-06T00:29:46.639738+0000","last_active":"2024-05-06T00:29:46.771675+0000","last_peered":"2024-05-06T00:29:46.771675+0000","last_clean":"2024-05-06T00:29:46.771675+0000","last_became_active":"2024-05-06T00:29:19.816383+0000","last_became_peered":"2024-05-06T00:29:19.816383+0000","last_unstale":"2024-05-06T00:29:46.771675+0000","last_undegraded":"2024-05-06T00:29:46.771675+0000","last_fullsized":"2024-05-06T00:29:46.771675+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:33:39.662072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003514901,"stat_sum":{"num_bytes":20261429,"num_objects":50,"num_object_clones":13,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":24,"num_read":13924,"num_read_kb":28056,"num_write":473,"num_write_kb":3425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.17","version":"159'303","reported_seq":4615,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.636821+0000","last_change":"2024-05-06T00:29:46.637150+0000","last_active":"2024-05-06T00:29:46.636821+0000","last_peered":"2024-05-06T00:29:46.636821+0000","last_clean":"2024-05-06T00:29:46.636821+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:29:46.636821+0000","last_undegraded":"2024-05-06T00:29:46.636821+0000","last_fullsized":"2024-05-06T00:29:46.636821+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:35:14.382141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016695449999999999,"stat_sum":{"num_bytes":66058851,"num_objects":54,"num_object_clones":10,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":29,"num_read":3631,"num_read_kb":19713,"num_write":269,"num_write_kb":5159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.16","version":"161'444","reported_seq":2560,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.687444+0000","last_change":"2024-05-06T00:29:46.687444+0000","last_active":"2024-05-06T00:29:46.687444+0000","last_peered":"2024-05-06T00:29:46.687444+0000","last_clean":"2024-05-06T00:29:46.687444+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:29:46.687444+0000","last_undegraded":"2024-05-06T00:29:46.687444+0000","last_fullsized":"2024-05-06T00:29:46.687444+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:52:24.528062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011215280000000001,"stat_sum":{"num_bytes":50453607,"num_objects":59,"num_object_clones":17,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":31,"num_read":2641,"num_read_kb":15138,"num_write":572,"num_write_kb":1907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.15","version":"159'264","reported_seq":5293,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.626825+0000","last_change":"2024-05-06T00:29:46.627281+0000","last_active":"2024-05-06T00:29:46.626825+0000","last_peered":"2024-05-06T00:29:46.626825+0000","last_clean":"2024-05-06T00:29:46.626825+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:29:46.626825+0000","last_undegraded":"2024-05-06T00:29:46.626825+0000","last_fullsized":"2024-05-06T00:29:46.626825+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:32:33.587916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093072000000000001,"stat_sum":{"num_bytes":12347904,"num_objects":47,"num_object_clones":14,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":27,"num_read":4880,"num_read_kb":11906,"num_write":258,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.14","version":"161'489","reported_seq":7714,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.763157+0000","last_change":"2024-05-06T00:29:46.628301+0000","last_active":"2024-05-06T00:29:46.763157+0000","last_peered":"2024-05-06T00:29:46.763157+0000","last_clean":"2024-05-06T00:29:46.763157+0000","last_became_active":"2024-05-06T00:28:31.620668+0000","last_became_peered":"2024-05-06T00:28:31.620668+0000","last_unstale":"2024-05-06T00:29:46.763157+0000","last_undegraded":"2024-05-06T00:29:46.763157+0000","last_fullsized":"2024-05-06T00:29:46.763157+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:34:02.136020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017863130000000001,"stat_sum":{"num_bytes":20864103,"num_objects":45,"num_object_clones":8,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":24,"num_read":7928,"num_read_kb":31359,"num_write":647,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.7","version":"0'0","reported_seq":250,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.626706+0000","last_change":"2024-05-06T00:28:27.989114+0000","last_active":"2024-05-06T00:29:46.626706+0000","last_peered":"2024-05-06T00:29:46.626706+0000","last_clean":"2024-05-06T00:29:46.626706+0000","last_became_active":"2024-05-06T00:28:27.988260+0000","last_became_peered":"2024-05-06T00:28:27.988260+0000","last_unstale":"2024-05-06T00:29:46.626706+0000","last_undegraded":"2024-05-06T00:29:46.626706+0000","last_fullsized":"2024-05-06T00:29:46.626706+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:19:42.564014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"161'861","reported_seq":7286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.716337+0000","last_change":"2024-05-06T00:29:46.636149+0000","last_active":"2024-05-06T00:29:46.716337+0000","last_peered":"2024-05-06T00:29:46.716337+0000","last_clean":"2024-05-06T00:29:46.716337+0000","last_became_active":"2024-05-06T00:28:06.816588+0000","last_became_peered":"2024-05-06T00:28:06.816588+0000","last_unstale":"2024-05-06T00:29:46.716337+0000","last_undegraded":"2024-05-06T00:29:46.716337+0000","last_fullsized":"2024-05-06T00:29:46.716337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":861,"log_dups_size":0,"ondisk_log_size":861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:18:37.732101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001139592,"stat_sum":{"num_bytes":33275510,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":21,"num_read":10368,"num_read_kb":29486,"num_write":1050,"num_write_kb":7368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.6","version":"0'0","reported_seq":311,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730639+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:29:46.730639+0000","last_peered":"2024-05-06T00:29:46.730639+0000","last_clean":"2024-05-06T00:29:46.730639+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:29:46.730639+0000","last_undegraded":"2024-05-06T00:29:46.730639+0000","last_fullsized":"2024-05-06T00:29:46.730639+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"160'652","reported_seq":6188,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.789170+0000","last_change":"2024-05-06T00:29:46.732649+0000","last_active":"2024-05-06T00:29:46.789170+0000","last_peered":"2024-05-06T00:29:46.789170+0000","last_clean":"2024-05-06T00:29:46.789170+0000","last_became_active":"2024-05-06T00:29:19.817230+0000","last_became_peered":"2024-05-06T00:29:19.817230+0000","last_unstale":"2024-05-06T00:29:46.789170+0000","last_undegraded":"2024-05-06T00:29:46.789170+0000","last_fullsized":"2024-05-06T00:29:46.789170+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:43:29.198439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001798577,"stat_sum":{"num_bytes":29034631,"num_objects":54,"num_object_clones":12,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":26,"num_read":8389,"num_read_kb":19353,"num_write":632,"num_write_kb":2764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.5","version":"0'0","reported_seq":180,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.635273+0000","last_change":"2024-05-06T00:28:27.997043+0000","last_active":"2024-05-06T00:29:46.635273+0000","last_peered":"2024-05-06T00:29:46.635273+0000","last_clean":"2024-05-06T00:29:46.635273+0000","last_became_active":"2024-05-06T00:28:27.996051+0000","last_became_peered":"2024-05-06T00:28:27.996051+0000","last_unstale":"2024-05-06T00:29:46.635273+0000","last_undegraded":"2024-05-06T00:29:46.635273+0000","last_fullsized":"2024-05-06T00:29:46.635273+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:03:48.195523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"160'558","reported_seq":8909,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.732068+0000","last_change":"2024-05-06T00:29:46.732372+0000","last_active":"2024-05-06T00:29:46.732068+0000","last_peered":"2024-05-06T00:29:46.732068+0000","last_clean":"2024-05-06T00:29:46.732068+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:29:46.732068+0000","last_undegraded":"2024-05-06T00:29:46.732068+0000","last_fullsized":"2024-05-06T00:29:46.732068+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:35:39.622026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018425539999999999,"stat_sum":{"num_bytes":48513137,"num_objects":71,"num_object_clones":14,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":37,"num_read":9571,"num_read_kb":30504,"num_write":634,"num_write_kb":5173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.4","version":"0'0","reported_seq":311,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730298+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:29:46.730298+0000","last_peered":"2024-05-06T00:29:46.730298+0000","last_clean":"2024-05-06T00:29:46.730298+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:29:46.730298+0000","last_undegraded":"2024-05-06T00:29:46.730298+0000","last_fullsized":"2024-05-06T00:29:46.730298+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"156'500","reported_seq":5274,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.686441+0000","last_change":"2024-05-06T00:29:46.687064+0000","last_active":"2024-05-06T00:29:46.686441+0000","last_peered":"2024-05-06T00:29:46.686441+0000","last_clean":"2024-05-06T00:29:46.686441+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:29:46.686441+0000","last_undegraded":"2024-05-06T00:29:46.686441+0000","last_fullsized":"2024-05-06T00:29:46.686441+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:30:47.592965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001161973,"stat_sum":{"num_bytes":20487803,"num_objects":40,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":20,"num_read":6385,"num_read_kb":16065,"num_write":445,"num_write_kb":2249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.3","version":"161'384","reported_seq":3714,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.780951+0000","last_change":"2024-05-06T00:29:46.627656+0000","last_active":"2024-05-06T00:29:46.780951+0000","last_peered":"2024-05-06T00:29:46.780951+0000","last_clean":"2024-05-06T00:29:46.780951+0000","last_became_active":"2024-05-06T00:28:02.782051+0000","last_became_peered":"2024-05-06T00:28:02.782051+0000","last_unstale":"2024-05-06T00:29:46.780951+0000","last_undegraded":"2024-05-06T00:29:46.780951+0000","last_fullsized":"2024-05-06T00:29:46.780951+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:23:23.026945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011819090000000001,"stat_sum":{"num_bytes":23907931,"num_objects":44,"num_object_clones":9,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":26,"num_read":3304,"num_read_kb":25318,"num_write":412,"num_write_kb":2647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.2","version":"0'0","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730592+0000","last_change":"2024-05-06T00:29:07.737006+0000","last_active":"2024-05-06T00:29:46.730592+0000","last_peered":"2024-05-06T00:29:46.730592+0000","last_clean":"2024-05-06T00:29:46.730592+0000","last_became_active":"2024-05-06T00:29:07.736607+0000","last_became_peered":"2024-05-06T00:29:07.736607+0000","last_unstale":"2024-05-06T00:29:46.730592+0000","last_undegraded":"2024-05-06T00:29:46.730592+0000","last_fullsized":"2024-05-06T00:29:46.730592+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:33:39.712007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"160'594","reported_seq":6533,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.731136+0000","last_change":"2024-05-06T00:29:46.731819+0000","last_active":"2024-05-06T00:29:46.731136+0000","last_peered":"2024-05-06T00:29:46.731136+0000","last_clean":"2024-05-06T00:29:46.731136+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:29:46.731136+0000","last_undegraded":"2024-05-06T00:29:46.731136+0000","last_fullsized":"2024-05-06T00:29:46.731136+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:32:47.123570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015726360000000001,"stat_sum":{"num_bytes":23576198,"num_objects":47,"num_object_clones":8,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":23,"num_read":7059,"num_read_kb":27041,"num_write":728,"num_write_kb":4737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.1","version":"0'0","reported_seq":75,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730370+0000","last_change":"2024-05-06T00:29:26.848406+0000","last_active":"2024-05-06T00:29:46.730370+0000","last_peered":"2024-05-06T00:29:46.730370+0000","last_clean":"2024-05-06T00:29:46.730370+0000","last_became_active":"2024-05-06T00:29:26.847624+0000","last_became_peered":"2024-05-06T00:29:26.847624+0000","last_unstale":"2024-05-06T00:29:46.730370+0000","last_undegraded":"2024-05-06T00:29:46.730370+0000","last_fullsized":"2024-05-06T00:29:46.730370+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:37:34.416843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"161'332","reported_seq":4662,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.824129+0000","last_change":"2024-05-06T00:29:46.639678+0000","last_active":"2024-05-06T00:29:46.824129+0000","last_peered":"2024-05-06T00:29:46.824129+0000","last_clean":"2024-05-06T00:29:46.824129+0000","last_became_active":"2024-05-06T00:28:05.820367+0000","last_became_peered":"2024-05-06T00:28:05.820367+0000","last_unstale":"2024-05-06T00:29:46.824129+0000","last_undegraded":"2024-05-06T00:29:46.824129+0000","last_fullsized":"2024-05-06T00:29:46.824129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:28:01.959176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030500760000000001,"stat_sum":{"num_bytes":43512366,"num_objects":55,"num_object_clones":12,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":25,"num_read":4009,"num_read_kb":14939,"num_write":343,"num_write_kb":2935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.0","version":"0'0","reported_seq":250,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730285+0000","last_change":"2024-05-06T00:28:27.993179+0000","last_active":"2024-05-06T00:29:46.730285+0000","last_peered":"2024-05-06T00:29:46.730285+0000","last_clean":"2024-05-06T00:29:46.730285+0000","last_became_active":"2024-05-06T00:28:27.991961+0000","last_became_peered":"2024-05-06T00:28:27.991961+0000","last_unstale":"2024-05-06T00:29:46.730285+0000","last_undegraded":"2024-05-06T00:29:46.730285+0000","last_fullsized":"2024-05-06T00:29:46.730285+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:21:51.936053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"159'729","reported_seq":10061,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.731719+0000","last_change":"2024-05-06T00:29:46.732019+0000","last_active":"2024-05-06T00:29:46.731719+0000","last_peered":"2024-05-06T00:29:46.731719+0000","last_clean":"2024-05-06T00:29:46.731719+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:29:46.731719+0000","last_undegraded":"2024-05-06T00:29:46.731719+0000","last_fullsized":"2024-05-06T00:29:46.731719+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:45:53.719163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017493789999999999,"stat_sum":{"num_bytes":18697290,"num_objects":42,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":23,"num_read":11952,"num_read_kb":24554,"num_write":850,"num_write_kb":3130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.3","version":"0'0","reported_seq":148,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.685978+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:29:46.685978+0000","last_peered":"2024-05-06T00:29:46.685978+0000","last_clean":"2024-05-06T00:29:46.685978+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:29:46.685978+0000","last_undegraded":"2024-05-06T00:29:46.685978+0000","last_fullsized":"2024-05-06T00:29:46.685978+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":198,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.634740+0000","last_change":"2024-05-06T00:28:28.005137+0000","last_active":"2024-05-06T00:29:46.634740+0000","last_peered":"2024-05-06T00:29:46.634740+0000","last_clean":"2024-05-06T00:29:46.634740+0000","last_became_active":"2024-05-06T00:28:27.995351+0000","last_became_peered":"2024-05-06T00:28:27.995351+0000","last_unstale":"2024-05-06T00:29:46.634740+0000","last_undegraded":"2024-05-06T00:29:46.634740+0000","last_fullsized":"2024-05-06T00:29:46.634740+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:17:37.661069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"161'550","reported_seq":5844,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.828860+0000","last_change":"2024-05-06T00:29:46.731395+0000","last_active":"2024-05-06T00:29:46.828860+0000","last_peered":"2024-05-06T00:29:46.828860+0000","last_clean":"2024-05-06T00:29:46.828860+0000","last_became_active":"2024-05-06T00:29:19.818623+0000","last_became_peered":"2024-05-06T00:29:19.818623+0000","last_unstale":"2024-05-06T00:29:46.828860+0000","last_undegraded":"2024-05-06T00:29:46.828860+0000","last_fullsized":"2024-05-06T00:29:46.828860+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:20:07.810561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013855339999999999,"stat_sum":{"num_bytes":27364536,"num_objects":61,"num_object_clones":12,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":35,"num_read":6326,"num_read_kb":27766,"num_write":569,"num_write_kb":8144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.9","version":"160'523","reported_seq":5564,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.628307+0000","last_change":"2024-05-06T00:29:46.628694+0000","last_active":"2024-05-06T00:29:46.628307+0000","last_peered":"2024-05-06T00:29:46.628307+0000","last_clean":"2024-05-06T00:29:46.628307+0000","last_became_active":"2024-05-06T00:28:58.673289+0000","last_became_peered":"2024-05-06T00:28:58.673289+0000","last_unstale":"2024-05-06T00:29:46.628307+0000","last_undegraded":"2024-05-06T00:29:46.628307+0000","last_fullsized":"2024-05-06T00:29:46.628307+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:39:54.026373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023129959999999999,"stat_sum":{"num_bytes":33241854,"num_objects":60,"num_object_clones":13,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":32,"num_read":5454,"num_read_kb":27751,"num_write":659,"num_write_kb":13296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.a","version":"161'716","reported_seq":8281,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.628210+0000","last_change":"2024-05-06T00:29:46.628210+0000","last_active":"2024-05-06T00:29:46.628210+0000","last_peered":"2024-05-06T00:29:46.628210+0000","last_clean":"2024-05-06T00:29:46.628210+0000","last_became_active":"2024-05-06T00:28:32.626303+0000","last_became_peered":"2024-05-06T00:28:32.626303+0000","last_unstale":"2024-05-06T00:29:46.628210+0000","last_undegraded":"2024-05-06T00:29:46.628210+0000","last_fullsized":"2024-05-06T00:29:46.628210+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:42:11.403523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001742513,"stat_sum":{"num_bytes":40936198,"num_objects":72,"num_object_clones":16,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":37,"num_read":9338,"num_read_kb":44686,"num_write":714,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.b","version":"160'313","reported_seq":6846,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.732361+0000","last_change":"2024-05-06T00:29:46.732741+0000","last_active":"2024-05-06T00:29:46.732361+0000","last_peered":"2024-05-06T00:29:46.732361+0000","last_clean":"2024-05-06T00:29:46.732361+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:29:46.732361+0000","last_undegraded":"2024-05-06T00:29:46.732361+0000","last_fullsized":"2024-05-06T00:29:46.732361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:37:05.212865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019875000000000001,"stat_sum":{"num_bytes":23610102,"num_objects":46,"num_object_clones":9,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":26,"num_read":5009,"num_read_kb":27735,"num_write":330,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.c","version":"161'378","reported_seq":6530,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.635949+0000","last_change":"2024-05-06T00:29:46.635949+0000","last_active":"2024-05-06T00:29:46.635949+0000","last_peered":"2024-05-06T00:29:46.635949+0000","last_clean":"2024-05-06T00:29:46.635949+0000","last_became_active":"2024-05-06T00:29:07.734795+0000","last_became_peered":"2024-05-06T00:29:07.734795+0000","last_unstale":"2024-05-06T00:29:46.635949+0000","last_undegraded":"2024-05-06T00:29:46.635949+0000","last_fullsized":"2024-05-06T00:29:46.635949+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:17:00.898748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001000756,"stat_sum":{"num_bytes":13232214,"num_objects":43,"num_object_clones":5,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":27,"num_read":5933,"num_read_kb":54215,"num_write":343,"num_write_kb":8900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.d","version":"159'838","reported_seq":12402,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.624839+0000","last_change":"2024-05-06T00:29:46.625054+0000","last_active":"2024-05-06T00:29:46.624839+0000","last_peered":"2024-05-06T00:29:46.624839+0000","last_clean":"2024-05-06T00:29:46.624839+0000","last_became_active":"2024-05-06T00:29:19.811713+0000","last_became_peered":"2024-05-06T00:29:19.811713+0000","last_unstale":"2024-05-06T00:29:46.624839+0000","last_undegraded":"2024-05-06T00:29:46.624839+0000","last_fullsized":"2024-05-06T00:29:46.624839+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:27:04.615260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074093800000000001,"stat_sum":{"num_bytes":22170697,"num_objects":67,"num_object_clones":15,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":41,"num_read":13471,"num_read_kb":33351,"num_write":1137,"num_write_kb":4649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.e","version":"161'772","reported_seq":12192,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.826684+0000","last_change":"2024-05-06T00:29:46.628720+0000","last_active":"2024-05-06T00:29:46.826684+0000","last_peered":"2024-05-06T00:29:46.826684+0000","last_clean":"2024-05-06T00:29:46.826684+0000","last_became_active":"2024-05-06T00:29:07.735487+0000","last_became_peered":"2024-05-06T00:29:07.735487+0000","last_unstale":"2024-05-06T00:29:46.826684+0000","last_undegraded":"2024-05-06T00:29:46.826684+0000","last_fullsized":"2024-05-06T00:29:46.826684+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":772,"log_dups_size":0,"ondisk_log_size":772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:00:02.183037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017950850000000001,"stat_sum":{"num_bytes":35188878,"num_objects":65,"num_object_clones":15,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":34,"num_read":14829,"num_read_kb":31441,"num_write":1000,"num_write_kb":4622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.f","version":"160'312","reported_seq":3955,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.687201+0000","last_change":"2024-05-06T00:29:46.687449+0000","last_active":"2024-05-06T00:29:46.687201+0000","last_peered":"2024-05-06T00:29:46.687201+0000","last_clean":"2024-05-06T00:29:46.687201+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:29:46.687201+0000","last_undegraded":"2024-05-06T00:29:46.687201+0000","last_fullsized":"2024-05-06T00:29:46.687201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:11:36.864651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077889500000000004,"stat_sum":{"num_bytes":13018595,"num_objects":44,"num_object_clones":8,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":25,"num_read":3461,"num_read_kb":19322,"num_write":358,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.10","version":"160'696","reported_seq":11198,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.636753+0000","last_change":"2024-05-06T00:29:46.637081+0000","last_active":"2024-05-06T00:29:46.636753+0000","last_peered":"2024-05-06T00:29:46.636753+0000","last_clean":"2024-05-06T00:29:46.636753+0000","last_became_active":"2024-05-06T00:29:07.734992+0000","last_became_peered":"2024-05-06T00:29:07.734992+0000","last_unstale":"2024-05-06T00:29:46.636753+0000","last_undegraded":"2024-05-06T00:29:46.636753+0000","last_fullsized":"2024-05-06T00:29:46.636753+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:34:11.779309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016722379999999999,"stat_sum":{"num_bytes":30602272,"num_objects":50,"num_object_clones":7,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":26,"num_read":12768,"num_read_kb":31552,"num_write":668,"num_write_kb":4616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.11","version":"158'376","reported_seq":5499,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.626524+0000","last_change":"2024-05-06T00:29:46.626831+0000","last_active":"2024-05-06T00:29:46.626524+0000","last_peered":"2024-05-06T00:29:46.626524+0000","last_clean":"2024-05-06T00:29:46.626524+0000","last_became_active":"2024-05-06T00:28:31.036279+0000","last_became_peered":"2024-05-06T00:28:31.036279+0000","last_unstale":"2024-05-06T00:29:46.626524+0000","last_undegraded":"2024-05-06T00:29:46.626524+0000","last_fullsized":"2024-05-06T00:29:46.626524+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:10:01.634119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059848200000000001,"stat_sum":{"num_bytes":31085688,"num_objects":54,"num_object_clones":8,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":34,"num_read":5317,"num_read_kb":10680,"num_write":492,"num_write_kb":3598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.12","version":"161'406","reported_seq":8730,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.636337+0000","last_change":"2024-05-06T00:29:46.636337+0000","last_active":"2024-05-06T00:29:46.636337+0000","last_peered":"2024-05-06T00:29:46.636337+0000","last_clean":"2024-05-06T00:29:46.636337+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:29:46.636337+0000","last_undegraded":"2024-05-06T00:29:46.636337+0000","last_fullsized":"2024-05-06T00:29:46.636337+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:49:22.049985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013278599999999999,"stat_sum":{"num_bytes":45947392,"num_objects":74,"num_object_clones":18,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":43,"num_read":8527,"num_read_kb":22946,"num_write":360,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.13","version":"160'570","reported_seq":8882,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.686137+0000","last_change":"2024-05-06T00:29:46.686744+0000","last_active":"2024-05-06T00:29:46.686137+0000","last_peered":"2024-05-06T00:29:46.686137+0000","last_clean":"2024-05-06T00:29:46.686137+0000","last_became_active":"2024-05-06T00:28:31.032323+0000","last_became_peered":"2024-05-06T00:28:31.032323+0000","last_unstale":"2024-05-06T00:29:46.686137+0000","last_undegraded":"2024-05-06T00:29:46.686137+0000","last_fullsized":"2024-05-06T00:29:46.686137+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:50:46.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081511400000000003,"stat_sum":{"num_bytes":38481505,"num_objects":50,"num_object_clones":11,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":21,"num_read":10053,"num_read_kb":40071,"num_write":553,"num_write_kb":5717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1033193875,"num_objects":1750,"num_object_clones":374,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":918,"num_read":255834,"num_read_kb":876278,"num_write":18500,"num_write_kb":192220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5546,"num_bytes_recovered":71938434,"num_keys_recovered":904,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":80322560,"data_stored":107239206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17320,"ondisk_log_size":17320,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640301,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65052,"kb_used_data":11652,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94306788,"statfs":{"total":96636764160,"available":96570150912,"internally_reserved":0,"allocated":11931648,"data_stored":12905336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640299,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84184,"kb_used_data":21064,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94287656,"statfs":{"total":96636764160,"available":96550559744,"internally_reserved":0,"allocated":21569536,"data_stored":28825237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72672,"kb_used_data":15116,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94299168,"statfs":{"total":96636764160,"available":96562348032,"internally_reserved":0,"allocated":15478784,"data_stored":18454842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72644,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94299196,"statfs":{"total":96636764160,"available":96562376704,"internally_reserved":0,"allocated":15458304,"data_stored":18499559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640301,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73280,"kb_used_data":15732,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94298560,"statfs":{"total":96636764160,"available":96561725440,"internally_reserved":0,"allocated":16109568,"data_stored":21014182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640299,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69836,"kb_used_data":12292,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94302004,"statfs":{"total":96636764160,"available":96565252096,"internally_reserved":0,"allocated":12587008,"data_stored":13603762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10063872,"data_stored":12208119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14180352,"data_stored":20208907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13529088,"data_stored":17694284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12955648,"data_stored":17059199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19640320,"data_stored":28019962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9953280,"data_stored":12048735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:29:49.055 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:29:49.096 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:49.096+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:49.122 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:49.122 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:49.136 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:49.184 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:49.185+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:49.254 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:49.255 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:49.258 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:49.257+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:49.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:49.359 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:49.357+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:29:49.380 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:29:49.398 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-05-06T00:29:48.674901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1033784243,"num_objects":1752,"num_object_clones":374,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":918,"num_read":255900,"num_read_kb":876333,"num_write":18601,"num_write_kb":193848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5554,"num_bytes_recovered":74299906,"num_keys_recovered":904,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17392,"ondisk_log_size":17392,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":437668,"kb_used_data":90952,"kb_used_omap":9,"kb_used_meta":346614,"kb_avail":565793372,"statfs":{"total":579820584960,"available":579372412928,"internally_reserved":0,"allocated":93134848,"data_stored":113302918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":354933408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-156482204,"num_objects":-139,"num_object_clones":-42,"num_object_copies":-278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-139,"num_whiteouts":-32,"num_read":1335,"num_read_kb":1141,"num_write":239,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.042335"},"pg_stats":[{"pgid":"3.1f","version":"160'604","reported_seq":12600,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.636280+0000","last_change":"2024-05-06T00:29:46.636553+0000","last_active":"2024-05-06T00:29:46.636280+0000","last_peered":"2024-05-06T00:29:46.636280+0000","last_clean":"2024-05-06T00:29:46.636280+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:29:46.636280+0000","last_undegraded":"2024-05-06T00:29:46.636280+0000","last_fullsized":"2024-05-06T00:29:46.636280+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:56:54.364195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001396659,"stat_sum":{"num_bytes":26140807,"num_objects":52,"num_object_clones":9,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":27,"num_read":12188,"num_read_kb":39835,"num_write":483,"num_write_kb":2565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1e","version":"160'310","reported_seq":9643,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.629362+0000","last_change":"2024-05-06T00:29:46.630009+0000","last_active":"2024-05-06T00:29:46.629362+0000","last_peered":"2024-05-06T00:29:46.629362+0000","last_clean":"2024-05-06T00:29:46.629362+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:29:46.629362+0000","last_undegraded":"2024-05-06T00:29:46.629362+0000","last_fullsized":"2024-05-06T00:29:46.629362+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:22:34.074033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011943030000000001,"stat_sum":{"num_bytes":73594399,"num_objects":66,"num_object_clones":19,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":28,"num_read":8756,"num_read_kb":50017,"num_write":277,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1d","version":"160'418","reported_seq":3693,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.687680+0000","last_change":"2024-05-06T00:29:46.688167+0000","last_active":"2024-05-06T00:29:46.687680+0000","last_peered":"2024-05-06T00:29:46.687680+0000","last_clean":"2024-05-06T00:29:46.687680+0000","last_became_active":"2024-05-06T00:29:13.784483+0000","last_became_peered":"2024-05-06T00:29:13.784483+0000","last_unstale":"2024-05-06T00:29:46.687680+0000","last_undegraded":"2024-05-06T00:29:46.687680+0000","last_fullsized":"2024-05-06T00:29:46.687680+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:32:50.803098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013667620000000001,"stat_sum":{"num_bytes":14054436,"num_objects":50,"num_object_clones":10,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":33,"num_read":3547,"num_read_kb":12883,"num_write":385,"num_write_kb":3277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1c","version":"161'788","reported_seq":9052,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.717953+0000","last_change":"2024-05-06T00:29:46.629058+0000","last_active":"2024-05-06T00:29:46.717953+0000","last_peered":"2024-05-06T00:29:46.717953+0000","last_clean":"2024-05-06T00:29:46.717953+0000","last_became_active":"2024-05-06T00:29:07.735705+0000","last_became_peered":"2024-05-06T00:29:07.735705+0000","last_unstale":"2024-05-06T00:29:46.717953+0000","last_undegraded":"2024-05-06T00:29:46.717953+0000","last_fullsized":"2024-05-06T00:29:46.717953+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:11:45.659856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001649944,"stat_sum":{"num_bytes":39499876,"num_objects":56,"num_object_clones":15,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":25,"num_read":12158,"num_read_kb":33759,"num_write":691,"num_write_kb":5259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1b","version":"160'1029","reported_seq":9360,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.637244+0000","last_change":"2024-05-06T00:29:46.637604+0000","last_active":"2024-05-06T00:29:46.637244+0000","last_peered":"2024-05-06T00:29:46.637244+0000","last_clean":"2024-05-06T00:29:46.637244+0000","last_became_active":"2024-05-06T00:28:05.820651+0000","last_became_peered":"2024-05-06T00:28:05.820651+0000","last_unstale":"2024-05-06T00:29:46.637244+0000","last_undegraded":"2024-05-06T00:29:46.637244+0000","last_fullsized":"2024-05-06T00:29:46.637244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1029,"log_dups_size":0,"ondisk_log_size":1029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:02:45.348907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002148582,"stat_sum":{"num_bytes":60492909,"num_objects":86,"num_object_clones":25,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":42,"num_read":12921,"num_read_kb":32938,"num_write":1023,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.1a","version":"160'344","reported_seq":6667,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.798638+0000","last_change":"2024-05-06T00:29:46.627906+0000","last_active":"2024-05-06T00:29:46.798638+0000","last_peered":"2024-05-06T00:29:46.798638+0000","last_clean":"2024-05-06T00:29:46.798638+0000","last_became_active":"2024-05-06T00:26:43.706058+0000","last_became_peered":"2024-05-06T00:26:43.706058+0000","last_unstale":"2024-05-06T00:29:46.798638+0000","last_undegraded":"2024-05-06T00:29:46.798638+0000","last_fullsized":"2024-05-06T00:29:46.798638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:40:03.987219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001498105,"stat_sum":{"num_bytes":35118761,"num_objects":49,"num_object_clones":7,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":24,"num_read":6828,"num_read_kb":25214,"num_write":368,"num_write_kb":13634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.19","version":"160'611","reported_seq":3431,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.732005+0000","last_change":"2024-05-06T00:29:46.732277+0000","last_active":"2024-05-06T00:29:46.732005+0000","last_peered":"2024-05-06T00:29:46.732005+0000","last_clean":"2024-05-06T00:29:46.732005+0000","last_became_active":"2024-05-06T00:26:43.714046+0000","last_became_peered":"2024-05-06T00:26:43.714046+0000","last_unstale":"2024-05-06T00:29:46.732005+0000","last_undegraded":"2024-05-06T00:29:46.732005+0000","last_fullsized":"2024-05-06T00:29:46.732005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:40:27.203337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935834,"stat_sum":{"num_bytes":18421996,"num_objects":49,"num_object_clones":11,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":22,"num_read":4909,"num_read_kb":16684,"num_write":777,"num_write_kb":5645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.18","version":"161'656","reported_seq":14240,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.771675+0000","last_change":"2024-05-06T00:29:46.639738+0000","last_active":"2024-05-06T00:29:46.771675+0000","last_peered":"2024-05-06T00:29:46.771675+0000","last_clean":"2024-05-06T00:29:46.771675+0000","last_became_active":"2024-05-06T00:29:19.816383+0000","last_became_peered":"2024-05-06T00:29:19.816383+0000","last_unstale":"2024-05-06T00:29:46.771675+0000","last_undegraded":"2024-05-06T00:29:46.771675+0000","last_fullsized":"2024-05-06T00:29:46.771675+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:33:39.662072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003514901,"stat_sum":{"num_bytes":20261429,"num_objects":50,"num_object_clones":13,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":24,"num_read":13924,"num_read_kb":28056,"num_write":473,"num_write_kb":3425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.17","version":"159'303","reported_seq":4615,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.636821+0000","last_change":"2024-05-06T00:29:46.637150+0000","last_active":"2024-05-06T00:29:46.636821+0000","last_peered":"2024-05-06T00:29:46.636821+0000","last_clean":"2024-05-06T00:29:46.636821+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:29:46.636821+0000","last_undegraded":"2024-05-06T00:29:46.636821+0000","last_fullsized":"2024-05-06T00:29:46.636821+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:35:14.382141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016695449999999999,"stat_sum":{"num_bytes":66058851,"num_objects":54,"num_object_clones":10,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":29,"num_read":3631,"num_read_kb":19713,"num_write":269,"num_write_kb":5159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.16","version":"161'444","reported_seq":2560,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.687444+0000","last_change":"2024-05-06T00:29:46.687444+0000","last_active":"2024-05-06T00:29:46.687444+0000","last_peered":"2024-05-06T00:29:46.687444+0000","last_clean":"2024-05-06T00:29:46.687444+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:29:46.687444+0000","last_undegraded":"2024-05-06T00:29:46.687444+0000","last_fullsized":"2024-05-06T00:29:46.687444+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:52:24.528062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011215280000000001,"stat_sum":{"num_bytes":50453607,"num_objects":59,"num_object_clones":17,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":31,"num_read":2641,"num_read_kb":15138,"num_write":572,"num_write_kb":1907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.15","version":"159'264","reported_seq":5293,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.626825+0000","last_change":"2024-05-06T00:29:46.627281+0000","last_active":"2024-05-06T00:29:46.626825+0000","last_peered":"2024-05-06T00:29:46.626825+0000","last_clean":"2024-05-06T00:29:46.626825+0000","last_became_active":"2024-05-06T00:26:51.610498+0000","last_became_peered":"2024-05-06T00:26:51.610498+0000","last_unstale":"2024-05-06T00:29:46.626825+0000","last_undegraded":"2024-05-06T00:29:46.626825+0000","last_fullsized":"2024-05-06T00:29:46.626825+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:32:33.587916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093072000000000001,"stat_sum":{"num_bytes":12347904,"num_objects":47,"num_object_clones":14,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":27,"num_read":4880,"num_read_kb":11906,"num_write":258,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.14","version":"161'489","reported_seq":7714,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.763157+0000","last_change":"2024-05-06T00:29:46.628301+0000","last_active":"2024-05-06T00:29:46.763157+0000","last_peered":"2024-05-06T00:29:46.763157+0000","last_clean":"2024-05-06T00:29:46.763157+0000","last_became_active":"2024-05-06T00:28:31.620668+0000","last_became_peered":"2024-05-06T00:28:31.620668+0000","last_unstale":"2024-05-06T00:29:46.763157+0000","last_undegraded":"2024-05-06T00:29:46.763157+0000","last_fullsized":"2024-05-06T00:29:46.763157+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:34:02.136020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017863130000000001,"stat_sum":{"num_bytes":20864103,"num_objects":45,"num_object_clones":8,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":24,"num_read":7928,"num_read_kb":31359,"num_write":647,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.7","version":"0'0","reported_seq":250,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.626706+0000","last_change":"2024-05-06T00:28:27.989114+0000","last_active":"2024-05-06T00:29:46.626706+0000","last_peered":"2024-05-06T00:29:46.626706+0000","last_clean":"2024-05-06T00:29:46.626706+0000","last_became_active":"2024-05-06T00:28:27.988260+0000","last_became_peered":"2024-05-06T00:28:27.988260+0000","last_unstale":"2024-05-06T00:29:46.626706+0000","last_undegraded":"2024-05-06T00:29:46.626706+0000","last_fullsized":"2024-05-06T00:29:46.626706+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:19:42.564014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"161'861","reported_seq":7286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.716337+0000","last_change":"2024-05-06T00:29:46.636149+0000","last_active":"2024-05-06T00:29:46.716337+0000","last_peered":"2024-05-06T00:29:46.716337+0000","last_clean":"2024-05-06T00:29:46.716337+0000","last_became_active":"2024-05-06T00:28:06.816588+0000","last_became_peered":"2024-05-06T00:28:06.816588+0000","last_unstale":"2024-05-06T00:29:46.716337+0000","last_undegraded":"2024-05-06T00:29:46.716337+0000","last_fullsized":"2024-05-06T00:29:46.716337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":861,"log_dups_size":0,"ondisk_log_size":861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:18:37.732101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001139592,"stat_sum":{"num_bytes":33275510,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":21,"num_read":10368,"num_read_kb":29486,"num_write":1050,"num_write_kb":7368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.6","version":"0'0","reported_seq":311,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730639+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:29:46.730639+0000","last_peered":"2024-05-06T00:29:46.730639+0000","last_clean":"2024-05-06T00:29:46.730639+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:29:46.730639+0000","last_undegraded":"2024-05-06T00:29:46.730639+0000","last_fullsized":"2024-05-06T00:29:46.730639+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"160'652","reported_seq":6188,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.789170+0000","last_change":"2024-05-06T00:29:46.732649+0000","last_active":"2024-05-06T00:29:46.789170+0000","last_peered":"2024-05-06T00:29:46.789170+0000","last_clean":"2024-05-06T00:29:46.789170+0000","last_became_active":"2024-05-06T00:29:19.817230+0000","last_became_peered":"2024-05-06T00:29:19.817230+0000","last_unstale":"2024-05-06T00:29:46.789170+0000","last_undegraded":"2024-05-06T00:29:46.789170+0000","last_fullsized":"2024-05-06T00:29:46.789170+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:43:29.198439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001798577,"stat_sum":{"num_bytes":29034631,"num_objects":54,"num_object_clones":12,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":26,"num_read":8389,"num_read_kb":19353,"num_write":632,"num_write_kb":2764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.5","version":"0'0","reported_seq":180,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.635273+0000","last_change":"2024-05-06T00:28:27.997043+0000","last_active":"2024-05-06T00:29:46.635273+0000","last_peered":"2024-05-06T00:29:46.635273+0000","last_clean":"2024-05-06T00:29:46.635273+0000","last_became_active":"2024-05-06T00:28:27.996051+0000","last_became_peered":"2024-05-06T00:28:27.996051+0000","last_unstale":"2024-05-06T00:29:46.635273+0000","last_undegraded":"2024-05-06T00:29:46.635273+0000","last_fullsized":"2024-05-06T00:29:46.635273+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:03:48.195523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"160'558","reported_seq":8909,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.732068+0000","last_change":"2024-05-06T00:29:46.732372+0000","last_active":"2024-05-06T00:29:46.732068+0000","last_peered":"2024-05-06T00:29:46.732068+0000","last_clean":"2024-05-06T00:29:46.732068+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:29:46.732068+0000","last_undegraded":"2024-05-06T00:29:46.732068+0000","last_fullsized":"2024-05-06T00:29:46.732068+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:35:39.622026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018425539999999999,"stat_sum":{"num_bytes":48513137,"num_objects":71,"num_object_clones":14,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":37,"num_read":9571,"num_read_kb":30504,"num_write":634,"num_write_kb":5173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.4","version":"0'0","reported_seq":311,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730298+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:29:46.730298+0000","last_peered":"2024-05-06T00:29:46.730298+0000","last_clean":"2024-05-06T00:29:46.730298+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:29:46.730298+0000","last_undegraded":"2024-05-06T00:29:46.730298+0000","last_fullsized":"2024-05-06T00:29:46.730298+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"156'500","reported_seq":5274,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.686441+0000","last_change":"2024-05-06T00:29:46.687064+0000","last_active":"2024-05-06T00:29:46.686441+0000","last_peered":"2024-05-06T00:29:46.686441+0000","last_clean":"2024-05-06T00:29:46.686441+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:29:46.686441+0000","last_undegraded":"2024-05-06T00:29:46.686441+0000","last_fullsized":"2024-05-06T00:29:46.686441+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:30:47.592965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001161973,"stat_sum":{"num_bytes":20487803,"num_objects":40,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":20,"num_read":6385,"num_read_kb":16065,"num_write":445,"num_write_kb":2249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.3","version":"161'384","reported_seq":3714,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.780951+0000","last_change":"2024-05-06T00:29:46.627656+0000","last_active":"2024-05-06T00:29:46.780951+0000","last_peered":"2024-05-06T00:29:46.780951+0000","last_clean":"2024-05-06T00:29:46.780951+0000","last_became_active":"2024-05-06T00:28:02.782051+0000","last_became_peered":"2024-05-06T00:28:02.782051+0000","last_unstale":"2024-05-06T00:29:46.780951+0000","last_undegraded":"2024-05-06T00:29:46.780951+0000","last_fullsized":"2024-05-06T00:29:46.780951+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:23:23.026945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011819090000000001,"stat_sum":{"num_bytes":23907931,"num_objects":44,"num_object_clones":9,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":26,"num_read":3304,"num_read_kb":25318,"num_write":412,"num_write_kb":2647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.2","version":"0'0","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730592+0000","last_change":"2024-05-06T00:29:07.737006+0000","last_active":"2024-05-06T00:29:46.730592+0000","last_peered":"2024-05-06T00:29:46.730592+0000","last_clean":"2024-05-06T00:29:46.730592+0000","last_became_active":"2024-05-06T00:29:07.736607+0000","last_became_peered":"2024-05-06T00:29:07.736607+0000","last_unstale":"2024-05-06T00:29:46.730592+0000","last_undegraded":"2024-05-06T00:29:46.730592+0000","last_fullsized":"2024-05-06T00:29:46.730592+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:33:39.712007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"160'594","reported_seq":6533,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.731136+0000","last_change":"2024-05-06T00:29:46.731819+0000","last_active":"2024-05-06T00:29:46.731136+0000","last_peered":"2024-05-06T00:29:46.731136+0000","last_clean":"2024-05-06T00:29:46.731136+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:29:46.731136+0000","last_undegraded":"2024-05-06T00:29:46.731136+0000","last_fullsized":"2024-05-06T00:29:46.731136+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:32:47.123570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015726360000000001,"stat_sum":{"num_bytes":23576198,"num_objects":47,"num_object_clones":8,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":23,"num_read":7059,"num_read_kb":27041,"num_write":728,"num_write_kb":4737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.1","version":"0'0","reported_seq":75,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730370+0000","last_change":"2024-05-06T00:29:26.848406+0000","last_active":"2024-05-06T00:29:46.730370+0000","last_peered":"2024-05-06T00:29:46.730370+0000","last_clean":"2024-05-06T00:29:46.730370+0000","last_became_active":"2024-05-06T00:29:26.847624+0000","last_became_peered":"2024-05-06T00:29:26.847624+0000","last_unstale":"2024-05-06T00:29:46.730370+0000","last_undegraded":"2024-05-06T00:29:46.730370+0000","last_fullsized":"2024-05-06T00:29:46.730370+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:37:34.416843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"161'332","reported_seq":4662,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.824129+0000","last_change":"2024-05-06T00:29:46.639678+0000","last_active":"2024-05-06T00:29:46.824129+0000","last_peered":"2024-05-06T00:29:46.824129+0000","last_clean":"2024-05-06T00:29:46.824129+0000","last_became_active":"2024-05-06T00:28:05.820367+0000","last_became_peered":"2024-05-06T00:28:05.820367+0000","last_unstale":"2024-05-06T00:29:46.824129+0000","last_undegraded":"2024-05-06T00:29:46.824129+0000","last_fullsized":"2024-05-06T00:29:46.824129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:28:01.959176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030500760000000001,"stat_sum":{"num_bytes":43512366,"num_objects":55,"num_object_clones":12,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":25,"num_read":4009,"num_read_kb":14939,"num_write":343,"num_write_kb":2935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.0","version":"0'0","reported_seq":250,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.730285+0000","last_change":"2024-05-06T00:28:27.993179+0000","last_active":"2024-05-06T00:29:46.730285+0000","last_peered":"2024-05-06T00:29:46.730285+0000","last_clean":"2024-05-06T00:29:46.730285+0000","last_became_active":"2024-05-06T00:28:27.991961+0000","last_became_peered":"2024-05-06T00:28:27.991961+0000","last_unstale":"2024-05-06T00:29:46.730285+0000","last_undegraded":"2024-05-06T00:29:46.730285+0000","last_fullsized":"2024-05-06T00:29:46.730285+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:21:51.936053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"159'729","reported_seq":10061,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.731719+0000","last_change":"2024-05-06T00:29:46.732019+0000","last_active":"2024-05-06T00:29:46.731719+0000","last_peered":"2024-05-06T00:29:46.731719+0000","last_clean":"2024-05-06T00:29:46.731719+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:29:46.731719+0000","last_undegraded":"2024-05-06T00:29:46.731719+0000","last_fullsized":"2024-05-06T00:29:46.731719+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:45:53.719163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017493789999999999,"stat_sum":{"num_bytes":18697290,"num_objects":42,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":23,"num_read":11952,"num_read_kb":24554,"num_write":850,"num_write_kb":3130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"2.3","version":"0'0","reported_seq":148,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.685978+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:29:46.685978+0000","last_peered":"2024-05-06T00:29:46.685978+0000","last_clean":"2024-05-06T00:29:46.685978+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:29:46.685978+0000","last_undegraded":"2024-05-06T00:29:46.685978+0000","last_fullsized":"2024-05-06T00:29:46.685978+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":198,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.634740+0000","last_change":"2024-05-06T00:28:28.005137+0000","last_active":"2024-05-06T00:29:46.634740+0000","last_peered":"2024-05-06T00:29:46.634740+0000","last_clean":"2024-05-06T00:29:46.634740+0000","last_became_active":"2024-05-06T00:28:27.995351+0000","last_became_peered":"2024-05-06T00:28:27.995351+0000","last_unstale":"2024-05-06T00:29:46.634740+0000","last_undegraded":"2024-05-06T00:29:46.634740+0000","last_fullsized":"2024-05-06T00:29:46.634740+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:17:37.661069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"161'550","reported_seq":5844,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.828860+0000","last_change":"2024-05-06T00:29:46.731395+0000","last_active":"2024-05-06T00:29:46.828860+0000","last_peered":"2024-05-06T00:29:46.828860+0000","last_clean":"2024-05-06T00:29:46.828860+0000","last_became_active":"2024-05-06T00:29:19.818623+0000","last_became_peered":"2024-05-06T00:29:19.818623+0000","last_unstale":"2024-05-06T00:29:46.828860+0000","last_undegraded":"2024-05-06T00:29:46.828860+0000","last_fullsized":"2024-05-06T00:29:46.828860+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:20:07.810561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013855339999999999,"stat_sum":{"num_bytes":27364536,"num_objects":61,"num_object_clones":12,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":35,"num_read":6326,"num_read_kb":27766,"num_write":569,"num_write_kb":8144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.9","version":"160'523","reported_seq":5564,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.628307+0000","last_change":"2024-05-06T00:29:46.628694+0000","last_active":"2024-05-06T00:29:46.628307+0000","last_peered":"2024-05-06T00:29:46.628307+0000","last_clean":"2024-05-06T00:29:46.628307+0000","last_became_active":"2024-05-06T00:28:58.673289+0000","last_became_peered":"2024-05-06T00:28:58.673289+0000","last_unstale":"2024-05-06T00:29:46.628307+0000","last_undegraded":"2024-05-06T00:29:46.628307+0000","last_fullsized":"2024-05-06T00:29:46.628307+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:39:54.026373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023129959999999999,"stat_sum":{"num_bytes":33241854,"num_objects":60,"num_object_clones":13,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":32,"num_read":5454,"num_read_kb":27751,"num_write":659,"num_write_kb":13296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.a","version":"161'716","reported_seq":8281,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.628210+0000","last_change":"2024-05-06T00:29:46.628210+0000","last_active":"2024-05-06T00:29:46.628210+0000","last_peered":"2024-05-06T00:29:46.628210+0000","last_clean":"2024-05-06T00:29:46.628210+0000","last_became_active":"2024-05-06T00:28:32.626303+0000","last_became_peered":"2024-05-06T00:28:32.626303+0000","last_unstale":"2024-05-06T00:29:46.628210+0000","last_undegraded":"2024-05-06T00:29:46.628210+0000","last_fullsized":"2024-05-06T00:29:46.628210+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:42:11.403523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001742513,"stat_sum":{"num_bytes":40936198,"num_objects":72,"num_object_clones":16,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":37,"num_read":9338,"num_read_kb":44686,"num_write":714,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.b","version":"160'313","reported_seq":6846,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.732361+0000","last_change":"2024-05-06T00:29:46.732741+0000","last_active":"2024-05-06T00:29:46.732361+0000","last_peered":"2024-05-06T00:29:46.732361+0000","last_clean":"2024-05-06T00:29:46.732361+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:29:46.732361+0000","last_undegraded":"2024-05-06T00:29:46.732361+0000","last_fullsized":"2024-05-06T00:29:46.732361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:37:05.212865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019875000000000001,"stat_sum":{"num_bytes":23610102,"num_objects":46,"num_object_clones":9,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":26,"num_read":5009,"num_read_kb":27735,"num_write":330,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.c","version":"161'378","reported_seq":6530,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.635949+0000","last_change":"2024-05-06T00:29:46.635949+0000","last_active":"2024-05-06T00:29:46.635949+0000","last_peered":"2024-05-06T00:29:46.635949+0000","last_clean":"2024-05-06T00:29:46.635949+0000","last_became_active":"2024-05-06T00:29:07.734795+0000","last_became_peered":"2024-05-06T00:29:07.734795+0000","last_unstale":"2024-05-06T00:29:46.635949+0000","last_undegraded":"2024-05-06T00:29:46.635949+0000","last_fullsized":"2024-05-06T00:29:46.635949+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:17:00.898748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001000756,"stat_sum":{"num_bytes":13232214,"num_objects":43,"num_object_clones":5,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":27,"num_read":5933,"num_read_kb":54215,"num_write":343,"num_write_kb":8900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.d","version":"159'838","reported_seq":12402,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.624839+0000","last_change":"2024-05-06T00:29:46.625054+0000","last_active":"2024-05-06T00:29:46.624839+0000","last_peered":"2024-05-06T00:29:46.624839+0000","last_clean":"2024-05-06T00:29:46.624839+0000","last_became_active":"2024-05-06T00:29:19.811713+0000","last_became_peered":"2024-05-06T00:29:19.811713+0000","last_unstale":"2024-05-06T00:29:46.624839+0000","last_undegraded":"2024-05-06T00:29:46.624839+0000","last_fullsized":"2024-05-06T00:29:46.624839+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:27:04.615260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074093800000000001,"stat_sum":{"num_bytes":22170697,"num_objects":67,"num_object_clones":15,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":41,"num_read":13471,"num_read_kb":33351,"num_write":1137,"num_write_kb":4649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.e","version":"161'772","reported_seq":12192,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.826684+0000","last_change":"2024-05-06T00:29:46.628720+0000","last_active":"2024-05-06T00:29:46.826684+0000","last_peered":"2024-05-06T00:29:46.826684+0000","last_clean":"2024-05-06T00:29:46.826684+0000","last_became_active":"2024-05-06T00:29:07.735487+0000","last_became_peered":"2024-05-06T00:29:07.735487+0000","last_unstale":"2024-05-06T00:29:46.826684+0000","last_undegraded":"2024-05-06T00:29:46.826684+0000","last_fullsized":"2024-05-06T00:29:46.826684+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":772,"log_dups_size":0,"ondisk_log_size":772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:00:02.183037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017950850000000001,"stat_sum":{"num_bytes":35188878,"num_objects":65,"num_object_clones":15,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":34,"num_read":14829,"num_read_kb":31441,"num_write":1000,"num_write_kb":4622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.f","version":"160'312","reported_seq":3955,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.687201+0000","last_change":"2024-05-06T00:29:46.687449+0000","last_active":"2024-05-06T00:29:46.687201+0000","last_peered":"2024-05-06T00:29:46.687201+0000","last_clean":"2024-05-06T00:29:46.687201+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:29:46.687201+0000","last_undegraded":"2024-05-06T00:29:46.687201+0000","last_fullsized":"2024-05-06T00:29:46.687201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:11:36.864651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077889500000000004,"stat_sum":{"num_bytes":13018595,"num_objects":44,"num_object_clones":8,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":25,"num_read":3461,"num_read_kb":19322,"num_write":358,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.10","version":"160'696","reported_seq":11198,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.636753+0000","last_change":"2024-05-06T00:29:46.637081+0000","last_active":"2024-05-06T00:29:46.636753+0000","last_peered":"2024-05-06T00:29:46.636753+0000","last_clean":"2024-05-06T00:29:46.636753+0000","last_became_active":"2024-05-06T00:29:07.734992+0000","last_became_peered":"2024-05-06T00:29:07.734992+0000","last_unstale":"2024-05-06T00:29:46.636753+0000","last_undegraded":"2024-05-06T00:29:46.636753+0000","last_fullsized":"2024-05-06T00:29:46.636753+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:34:11.779309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016722379999999999,"stat_sum":{"num_bytes":30602272,"num_objects":50,"num_object_clones":7,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":26,"num_read":12768,"num_read_kb":31552,"num_write":668,"num_write_kb":4616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.11","version":"158'376","reported_seq":5499,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.626524+0000","last_change":"2024-05-06T00:29:46.626831+0000","last_active":"2024-05-06T00:29:46.626524+0000","last_peered":"2024-05-06T00:29:46.626524+0000","last_clean":"2024-05-06T00:29:46.626524+0000","last_became_active":"2024-05-06T00:28:31.036279+0000","last_became_peered":"2024-05-06T00:28:31.036279+0000","last_unstale":"2024-05-06T00:29:46.626524+0000","last_undegraded":"2024-05-06T00:29:46.626524+0000","last_fullsized":"2024-05-06T00:29:46.626524+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:10:01.634119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059848200000000001,"stat_sum":{"num_bytes":31085688,"num_objects":54,"num_object_clones":8,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":34,"num_read":5317,"num_read_kb":10680,"num_write":492,"num_write_kb":3598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.12","version":"161'406","reported_seq":8730,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.636337+0000","last_change":"2024-05-06T00:29:46.636337+0000","last_active":"2024-05-06T00:29:46.636337+0000","last_peered":"2024-05-06T00:29:46.636337+0000","last_clean":"2024-05-06T00:29:46.636337+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:29:46.636337+0000","last_undegraded":"2024-05-06T00:29:46.636337+0000","last_fullsized":"2024-05-06T00:29:46.636337+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:49:22.049985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013278599999999999,"stat_sum":{"num_bytes":45947392,"num_objects":74,"num_object_clones":18,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":43,"num_read":8527,"num_read_kb":22946,"num_write":360,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"24","length":"5"}]},{"pgid":"3.13","version":"160'570","reported_seq":8882,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-06T00:29:46.686137+0000","last_change":"2024-05-06T00:29:46.686744+0000","last_active":"2024-05-06T00:29:46.686137+0000","last_peered":"2024-05-06T00:29:46.686137+0000","last_clean":"2024-05-06T00:29:46.686137+0000","last_became_active":"2024-05-06T00:28:31.032323+0000","last_became_peered":"2024-05-06T00:28:31.032323+0000","last_unstale":"2024-05-06T00:29:46.686137+0000","last_undegraded":"2024-05-06T00:29:46.686137+0000","last_fullsized":"2024-05-06T00:29:46.686137+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:50:46.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081511400000000003,"stat_sum":{"num_bytes":38481505,"num_objects":50,"num_object_clones":11,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":21,"num_read":10053,"num_read_kb":40071,"num_write":553,"num_write_kb":5717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1033193875,"num_objects":1750,"num_object_clones":374,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":918,"num_read":255834,"num_read_kb":876278,"num_write":18500,"num_write_kb":192220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5546,"num_bytes_recovered":71938434,"num_keys_recovered":904,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":80322560,"data_stored":107239206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17320,"ondisk_log_size":17320,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640301,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65052,"kb_used_data":11652,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94306788,"statfs":{"total":96636764160,"available":96570150912,"internally_reserved":0,"allocated":11931648,"data_stored":12905336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640299,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84184,"kb_used_data":21064,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94287656,"statfs":{"total":96636764160,"available":96550559744,"internally_reserved":0,"allocated":21569536,"data_stored":28825237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72672,"kb_used_data":15116,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94299168,"statfs":{"total":96636764160,"available":96562348032,"internally_reserved":0,"allocated":15478784,"data_stored":18454842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72644,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94299196,"statfs":{"total":96636764160,"available":96562376704,"internally_reserved":0,"allocated":15458304,"data_stored":18499559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640301,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73280,"kb_used_data":15732,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94298560,"statfs":{"total":96636764160,"available":96561725440,"internally_reserved":0,"allocated":16109568,"data_stored":21014182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640299,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69836,"kb_used_data":12292,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94302004,"statfs":{"total":96636764160,"available":96565252096,"internally_reserved":0,"allocated":12587008,"data_stored":13603762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10063872,"data_stored":12208119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14180352,"data_stored":20208907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13529088,"data_stored":17694284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12955648,"data_stored":17059199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19640320,"data_stored":28019962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9953280,"data_stored":12048735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:29:49.400 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-05-06T00:29:49.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:29:49.400 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-05-06T00:29:49.400 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-06T00:29:49.401 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-06T00:29:49.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-05-06T00:29:49.433 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:49.433 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:49.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:49.460 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:49.460+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:49.532 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:49.532 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "" 2024-05-06T00:29:49.532 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:49.560 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:49.560+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:49.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:49.662 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:49.660+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:29:49.749 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:49.749 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:49.762 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:49.761+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:49.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:49.863 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:49.860+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:49.941 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:49.941 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:49.956 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:50.000 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:50.000+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:50.066 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:50.064+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:50.086 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:50.086 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:50.101 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:50.196 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:50.196+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:50.267 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:50.267 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:50.268 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:50.268+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:50.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:50.369 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:50.369+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:50.445 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:50.445 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:50.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:50.504 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:50.504+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:50.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:50.576 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:50.576 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:50.577 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:50.576+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:50.593 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:50.678 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:50.678 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:50.676+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:50.698 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:50.698 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:50.717 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:50.718 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:50.761 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:50.770 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:50.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:50.770 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:50.770 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:50.771 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:50.772 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:50.769+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:50.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:50.784 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:50.786 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:50.892 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:50.892+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:50.974 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:50.974 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:50.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:50.974 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:50.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.974 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:50.974 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.946099656, 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:50.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:50.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:50.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.946078867, 2024-05-06T00:29:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:50.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:29:50.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:50.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.94605880500001, 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:50.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:29:50.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:29:50.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.946039529, 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:50.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.94601941099999, 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:50.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:29:50.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.94598800200001, 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:50.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:50.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.94595509, 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.005 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.945934503, 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.006 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:29:51.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:29:51.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:29:51.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.94591512300001, 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.010 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:29:51.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:51.014 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.945870442, 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:51.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:29:51.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:29:51.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.945805351, 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:29:51.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:29:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.945773777, 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:29:51.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:29:51.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.945751516, 2024-05-06T00:29:51.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:29:51.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:29:51.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.9457304, 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:29:51.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:29:51.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:29:51.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.037 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.037 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.037 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.28801377400001, 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.041 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.28760353200001, 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 59.348958938000003, 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:29:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:29:51.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:29:51.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 59.348887654000002, 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:51.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:29:51.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:29:51.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 59.348856021000003, 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:29:51.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:29:51.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 59.344637474999999, 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:29:51.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:29:51.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:29:51.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:29:51.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.067 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:51.068 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:50.973+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:51.069 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:51.074 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:51.073+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:51.175 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:51.172+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:51.238 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:51.239 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:51.239 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:51.239 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:51.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:51.296 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:51.296+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:51.377 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:51.376+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:51.421 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:51.421 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:51.436 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:51.480 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:51.480+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:51.579 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:51.577+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:51.606 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 143.59947522499999, 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:29:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:29:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:29:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 143.59924665299999, 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:29:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:29:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 143.59916936900001, 2024-05-06T00:29:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 143.59897338799999, 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:29:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:29:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 110.691864012, 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:29:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:29:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:29:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 110.69180313299999, 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:29:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:29:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.574246683, 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:29:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:29:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.57420195500001, 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:29:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.57416355399999, 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:29:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:29:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:29:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:29:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:29:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:29:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:29:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:29:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.574126292, 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:29:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.574089383, 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:29:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.574051465, 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:29:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.57401435, 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:29:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:29:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.568233901, 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:29:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:29:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.568195817, 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.56815814300001, 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:29:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.558722633, 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.55869020599999, 2024-05-06T00:29:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.558666324, 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:29:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.55864639399999, 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:29:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:51.694 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:51.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:51.696 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:51.692+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:51.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:29:51.796 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:51.797+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:51.836 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:51.836 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:51.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:51.897 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:51.896+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:51.998 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:51.997+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:52.032 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:52.032 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:52.032 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:52.032 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:52.032 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:52.046 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:52.098 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:52.096+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:52.199 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:52.197+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:52.218 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:52.218 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:52.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:52.218 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:52.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 202.61343842700001, 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:595 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 154.34075455199999, 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035125542000000003, 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 595 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.225 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875083+0000", 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875084+0000", 2024-05-06T00:29:52.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0299999999999998e-07 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875088+0000", 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6029999999999998e-06 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875145+0000", 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7040999999999998e-05 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875156+0000", 2024-05-06T00:29:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0475999999999999e-05 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906004+0000", 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030847856999999999 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906102+0000", 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8251000000000005e-05 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.906550+0000", 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00044823900000000002 2024-05-06T00:29:52.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910054+0000", 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035033030000000001 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910073+0000", 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9802000000000001e-05 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910132+0000", 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8604999999999999e-05 2024-05-06T00:29:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910209+0000", 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6562000000000003e-05 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 154.34066587300001, 2024-05-06T00:29:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:29:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:29:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 154.34056393, 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:29:52.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:29:52.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 154.34045482799999, 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:29:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:29:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 154.340356969, 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.254 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 154.34029191900001, 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.255 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:29:52.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:29:52.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:29:52.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 154.340199273, 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:29:52.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:29:52.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:29:52.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:29:52.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:29:52.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.267 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 154.34012726899999, 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:52.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:29:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:29:52.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.580665783, 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.273 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:29:52.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.580646535, 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.279 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:29:52.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.580596415, 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.290 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.580575364, 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:29:52.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:29:52.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:52.296 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.580560045, 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:29:52.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.58054316299999, 2024-05-06T00:29:52.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:29:52.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.580524827, 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.308 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:29:52.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.58050882500001, 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:29:52.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:29:52.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:29:52.319 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 153.580422462, 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:29:52.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:29:52.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:29:52.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [read 12582912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.401172215999999, 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857773900000002, 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.325 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:29:52.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.812763+0000", 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99789039000000002 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813015+0000", 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025184099999999999 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813243+0000", 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022833400000000001 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.330 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [sparse-read 12587008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.401071743999999, 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99922042799999999, 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813844+0000", 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99888496199999999 2024-05-06T00:29:52.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813901+0000", 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6561000000000003e-05 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813986+0000", 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5643999999999995e-05 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.337 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.337 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:52.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:52.339 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:52.337+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:52.439 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:52.440+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:52.507 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:52.507 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:52.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:52.541 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:52.540+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:52.641 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:52.640+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:52.644 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:52.644 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:52.644 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:52.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:52.709 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:29:52.743 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:52.740+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 131.1267397, 2024-05-06T00:29:52.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:52.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:52.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.147498195, 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.792 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:52.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.147423976, 2024-05-06T00:29:52.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:52.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:52.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.147362077, 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:52.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:52.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:52.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.147302478, 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:52.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:52.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.14721834300001, 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:52.809 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:52.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:52.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.813 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.14715283, 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:52.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:52.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.147094968, 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:52.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:52.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:52.822 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.146998373, 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:52.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.14692064800001, 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.146847862, 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.146770371, 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.146676242, 2024-05-06T00:29:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.14659328, 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.14651917499999, 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.146419986, 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.146356834, 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.146256569, 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.146167197, 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.900308514, 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:52.873 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:52.874 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:52.875 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:52.872+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:52.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:52.976 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:52.973+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:53.003 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:53.003 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:53.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:53.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:53.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:53.079 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:53.079 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:53.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:53.097 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:53.105 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:53.105+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:53.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:53.115 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:53.178 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:53.176+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:53.188 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:53.188 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:53.202 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:53.279 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:53.276+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:53.364 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:53.364 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:53.378 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:53.379 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:53.380+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:53.481 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:53.481+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:53.509 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:53.509 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:53.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:53.581 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:53.580+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:53.646 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:53.646 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:53.660 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:53.704 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:53.704+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:53.783 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:53.781+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:53.838 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:53.838 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:53.852 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:53.900 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:53.900+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:53.982 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:53.983 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:53.985 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:53.984+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:53.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:54.086 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:54.085+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:54.163 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:54.163 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:54.177 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:54.224 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:54.224+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:54.288 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:54.285+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:54.300 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:54.300 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:54.315 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:54.389 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:54.389+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:54.479 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:54.479 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:54.490 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:54.489+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:54.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:54.546 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:29:54.547 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:29:54.547 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-05-06T00:29:54.547 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-05-06T00:29:54.591 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:54.589+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:54.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:54.692 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:54.689+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:54.701 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:54.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-05-06T00:29:54.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:29:54.777 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:54.778 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:54.792 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:54.789+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:54.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:54.893 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:54.893+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:54.907 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:54.907 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-05-06T00:29:54.934 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:54.935 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:54.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:54.995 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:54.993+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:55.052 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:55.052 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:55.052 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:55.052 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:55.056 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:55.058 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-05-06T00:29:55.096 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:55.093+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:55.114 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:55.116 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:55.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:55.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:55.206 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:55.207 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-05-06T00:29:55.216 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:55.217+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:55.265 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:55.266 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:55.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:55.281 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:55.294 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:55.294 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:55.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:55.297 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:55.297+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:55.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:55.316 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:55.316 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:55.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:55.343 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:55.369 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:55.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-05-06T00:29:55.370 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:55.371 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:55.398 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:55.397+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:55.474 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:55.474 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:55.474 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:55.474 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:55.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:55.498 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:55.496+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-06T00:29:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-06T00:29:55.513 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-06T00:29:55.513 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-06T00:29:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:29:55.515 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-06T00:29:55.599 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:55.597+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:55.653 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:55.654 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:55.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:55.701 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:55.700+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:29:55.802 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:55.801+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4955.0:5837 3.1c 3:3da39b01:::rbd_data.110ce7c6f85.000000000000004b:head [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 134.190271481, 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96219569699999996, 2024-05-06T00:29:55.852 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4955", 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/769837695", 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5837 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660277+0000", 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5199999999999997e-07 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660280+0000", 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8559999999999998e-06 2024-05-06T00:29:55.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660286+0000", 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.835e-06 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660290+0000", 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.172e-06 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660369+0000", 2024-05-06T00:29:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8465000000000002e-05 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:41.660380+0000", 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622183+0000", 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96180299000000002 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:42.622473+0000", 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028914899999999999 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1315 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 414720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.21102997600001, 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094995498999999997, 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.857 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1315 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639518+0000", 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639520+0000", 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.393e-06 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639527+0000", 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8240000000000002e-06 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639544+0000", 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7723e-05 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639558+0000", 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3606e-05 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734100+0000", 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094542574000000004 2024-05-06T00:29:55.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734319+0000", 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000218316 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734514+0000", 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019506300000000001 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1316 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 420864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.21095575699999, 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354823000000005, 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1316 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.862 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639593+0000", 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639594+0000", 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2529999999999999e-06 2024-05-06T00:29:55.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639599+0000", 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.507e-06 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639621+0000", 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2042999999999999e-05 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639635+0000", 2024-05-06T00:29:55.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3913e-05 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734653+0000", 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095017292000000003 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734811+0000", 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015864699999999999 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.734947+0000", 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013616799999999999 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1317 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 427008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.21089385800001, 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095823338999999993, 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1317 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639654+0000", 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639656+0000", 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.19e-06 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639661+0000", 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639683+0000", 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1483000000000001e-05 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639696+0000", 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3516000000000001e-05 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735050+0000", 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095354381000000002 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735280+0000", 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000229874 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735478+0000", 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019746499999999999 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1318 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 433152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.21083425899999, 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096300127999999999, 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1318 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639714+0000", 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639715+0000", 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.116e-06 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639721+0000", 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3700000000000003e-06 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639738+0000", 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7104000000000001e-05 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639774+0000", 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6699000000000002e-05 2024-05-06T00:29:55.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735564+0000", 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095789787000000001 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.735813+0000", 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024838699999999998 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736014+0000", 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020166499999999999 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1319 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 439296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210750124, 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096716872999999995, 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1319 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639798+0000", 2024-05-06T00:29:55.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639799+0000", 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639805+0000", 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8760000000000002e-06 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639824+0000", 2024-05-06T00:29:55.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8995999999999999e-05 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639837+0000", 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3203e-05 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736104+0000", 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096266078000000005 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736316+0000", 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021284 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736515+0000", 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198636 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1320 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 445440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.879 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210684611, 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097170612000000003, 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1320 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639864+0000", 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639865+0000", 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.176e-06 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639870+0000", 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5230000000000003e-06 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639888+0000", 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7350000000000002e-05 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639901+0000", 2024-05-06T00:29:55.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736598+0000", 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096697569999999997 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.736834+0000", 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023613900000000001 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737034+0000", 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019988800000000001 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1321 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 451584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210626749, 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097555165999999999, 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1321 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639922+0000", 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639923+0000", 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.173e-06 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639928+0000", 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1800000000000004e-06 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639962+0000", 2024-05-06T00:29:55.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3983000000000003e-05 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.639997+0000", 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5568999999999998e-05 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737162+0000", 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097164902999999997 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737324+0000", 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016189399999999999 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737477+0000", 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152464 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1322 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 457728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210530154, 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098014614999999999, 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1322 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:55.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640018+0000", 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640019+0000", 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3260000000000001e-06 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640025+0000", 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7540000000000003e-06 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640043+0000", 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7846000000000002e-05 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640077+0000", 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3793e-05 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737610+0000", 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097532919999999995 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.737832+0000", 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022225700000000001 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738033+0000", 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000200719 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1323 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 463872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210452429, 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098340123000000002, 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1323 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640096+0000", 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640097+0000", 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.167e-06 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640103+0000", 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4970000000000001e-06 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640131+0000", 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8462e-05 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640146+0000", 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4966999999999999e-05 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738141+0000", 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097995107999999997 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738302+0000", 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016074700000000001 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738436+0000", 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134175 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1324 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 470016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210379643, 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098676202000000005, 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.897 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1324 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640169+0000", 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640170+0000", 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1930000000000001e-06 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640176+0000", 2024-05-06T00:29:55.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6899999999999997e-06 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640197+0000", 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268999999999999e-05 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640211+0000", 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.466e-05 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738633+0000", 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098421340999999996 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738750+0000", 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117088 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738845+0000", 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4961000000000001e-05 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1325 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 476160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210302152, 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098978366999999998, 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.902 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1325 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640246+0000", 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640247+0000", 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1540000000000001e-06 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640268+0000", 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1046000000000001e-05 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640304+0000", 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5562999999999998e-05 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640319+0000", 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5469999999999999e-05 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.738952+0000", 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098632144000000005 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739095+0000", 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014317400000000001 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739225+0000", 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012981599999999999 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1326 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 482304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.21020802300001, 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099110011999999997, 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.906 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1326 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:55.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640340+0000", 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640341+0000", 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.184e-06 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640348+0000", 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640368+0000", 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0109999999999999e-05 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640403+0000", 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4969999999999999e-05 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739274+0000", 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098871018000000005 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739378+0000", 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104168 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739450+0000", 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2373000000000007e-05 2024-05-06T00:29:55.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1327 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 488448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210125061, 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099210074999999995, 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1327 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640423+0000", 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640425+0000", 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.263e-06 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640431+0000", 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.127e-06 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640449+0000", 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8263000000000001e-05 2024-05-06T00:29:55.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640463+0000", 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3643e-05 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739496+0000", 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099033835000000001 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739566+0000", 2024-05-06T00:29:55.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9233999999999995e-05 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739633+0000", 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7710000000000001e-05 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1328 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 494592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.210050956, 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099342436000000006, 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.917 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1328 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640497+0000", 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640498+0000", 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1340000000000001e-06 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640505+0000", 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3160000000000002e-06 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640537+0000", 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2252999999999998e-05 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640560+0000", 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272999999999999e-05 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739684+0000", 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099124016999999995 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739773+0000", 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8194000000000003e-05 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739840+0000", 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7249000000000006e-05 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.921 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1329 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 500736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.20995176700001, 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099420156999999995, 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1329 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640597+0000", 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640598+0000", 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.15e-06 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640603+0000", 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640621+0000", 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7523000000000001e-05 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640637+0000", 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5937e-05 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739885+0000", 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099248749999999997 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.739952+0000", 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6773999999999995e-05 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740017+0000", 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4467000000000005e-05 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.925 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1330 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 506880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.209888615, 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099532575999999998, 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.926 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1330 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640660+0000", 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640661+0000", 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1170000000000001e-06 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640683+0000", 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1943e-05 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640710+0000", 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7608999999999999e-05 2024-05-06T00:29:55.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640725+0000", 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4686e-05 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740060+0000", 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099335246000000002 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740127+0000", 2024-05-06T00:29:55.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6570999999999997e-05 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740192+0000", 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5403999999999999e-05 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1331 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 513024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.20978835, 2024-05-06T00:29:55.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099627344000000007, 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1331 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640760+0000", 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640761+0000", 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640778+0000", 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6529e-05 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640799+0000", 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0953000000000001e-05 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640829+0000", 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9983000000000001e-05 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740234+0000", 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099405263999999993 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740305+0000", 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1470999999999994e-05 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740387+0000", 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.195e-05 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:1332 3.15 3:a9011deb:::rbd_data.110cf097386.000000000000000b:19 [read 519168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.20969897800001, 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099707360999999994, 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1332 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640849+0000", 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640851+0000", 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2860000000000001e-06 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640857+0000", 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9800000000000003e-06 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640877+0000", 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0466999999999999e-05 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.640893+0000", 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5622999999999999e-05 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740421+0000", 2024-05-06T00:29:55.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099527901000000002 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740490+0000", 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9082e-05 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.740557+0000", 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7021999999999996e-05 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5155.0:2236 3.11 3:8936265b:::rbd_data.110cf097386.0000000000000011:19 [read 1571840~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 115.963840295, 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095077437000000001, 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5155", 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2236 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:55.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886708+0000", 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886709+0000", 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0699999999999997e-07 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886712+0000", 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886720+0000", 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9360000000000005e-06 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.886728+0000", 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999996e-06 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981597+0000", 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094869973999999996 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981693+0000", 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5915999999999994e-05 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:59.981785+0000", 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2044000000000002e-05 2024-05-06T00:29:55.942 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:55.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:55.943 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:55.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:55.945 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:55.941+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:56.046 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:56.045+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:56.117 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:56.117 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:56.117 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:29:56.118 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:56.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:56.146 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:56.144+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:56.247 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:56.244+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:56.297 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:29:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:29:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:29:56.297 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:56.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:56.348 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:56.348+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:56.439 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:29:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:29:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:29:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:29:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.476573270000003, 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:29:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:29:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:29:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.475388655000003, 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:29:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.591967609999998, 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.591930396999999, 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:29:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:29:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.591841590999998, 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:29:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:29:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.591781398999998, 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:29:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.591727096, 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:29:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.591660456, 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:29:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:29:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.591598183999999, 2024-05-06T00:29:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:29:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:29:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.591539187999999, 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:29:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:29:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.585134467000003, 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:29:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:29:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.585085733, 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.585041480999998, 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.584970448999997, 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.584919634000002, 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:29:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:29:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:29:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.576180485999998, 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:29:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.576157471999998, 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.576135221999998, 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.576025371999997, 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:29:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:29:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:29:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.575961321999998, 2024-05-06T00:29:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:29:56.526 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:29:56.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:56.529 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:56.528+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:56.634 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:56.632+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:56.675 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:56.675 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:56.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(osd.0.0:4189 3.1b e171/83 ondisk, result = 0)", 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:29:56.677465+0000", 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.00014562000000000001, 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015019400000000001, 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677465+0000", 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:56.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677465+0000", 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677465+0000", 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3600000000000003e-07 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677469+0000", 2024-05-06T00:29:56.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0899999999999998e-06 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677482+0000", 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2486999999999999e-05 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677489+0000", 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.922e-06 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677535+0000", 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6424000000000001e-05 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677540+0000", 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7350000000000004e-06 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:56.677615+0000", 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4800000000000002e-05 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:56.679 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:56.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:56.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:56.680 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:56.680 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 1 2024-05-06T00:29:56.680 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:56.691 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:56.730 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:56.729+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:56.831 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:56.829+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:56.863 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:56.864 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:56.864 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:56.864 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:56.864 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:56.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:29:56.931 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:56.928+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 178.203399176, 2024-05-06T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:57.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:29:57.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 178.203382973, 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:29:57.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:29:57.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:29:57.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.195152756, 2024-05-06T00:29:57.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:29:57.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.195121386, 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:29:57.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.195087597, 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:29:57.032 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:29:57.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:29:57.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:29:57.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.194907141, 2024-05-06T00:29:57.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:57.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:29:57.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.042 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.19487349800001, 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.043 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:29:57.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:29:57.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:29:57.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:29:57.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:29:57.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:29:57.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:29:57.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.19484971599999, 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:29:57.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:29:57.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.194823439, 2024-05-06T00:29:57.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:29:57.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:29:57.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.059 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.194780198, 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:29:57.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.19475056900001, 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:29:57.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.194717719, 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:29:57.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.194686081, 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:29:57.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:29:57.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:57.080 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.19465327899999, 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.085 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.194621246, 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:29:57.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:29:57.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.194589167, 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:29:57.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:29:57.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:29:57.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.194557859, 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:29:57.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:29:57.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.354412142000001, 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:57.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:29:57.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:29:57.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.353005992999996, 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:29:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.352953034999999, 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:29:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:57.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:57.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:29:57.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:29:57.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:57.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:29:57.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:29:57.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:57.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:29:57.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:29:57.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:57.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:29:57.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:29:57.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:29:57.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:29:57.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:29:57.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:29:57.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:29:57.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:57.126 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:29:57.127 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:57.124+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:57.226 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:57.224+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:57.294 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:57.294 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:57.295 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:29:57.295 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:57.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:29:57.328 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:57.325+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:57.388 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:29:57.429 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:57.429+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:57.472 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:57.473 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:57.473 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:57.473 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:57.473 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:57.487 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:29:57.530 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:57.529+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:57.533 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:57.533 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:57.549 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:57.550 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:57.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:57.568 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:57.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:57.588 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:57.608 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:57.608 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:57.615 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:57.615 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:57.629 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:29:57.630 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:57.630 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:57.631 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:57.629+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:57.732 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:57.732+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:57.745 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:29:57.748 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:57.748 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:57.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:29:57.860 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:57.860+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:57.933 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:57.932+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:57.941 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:57.941 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:57.956 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:29:58.044 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:58.044+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:58.115 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:58.115 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:58.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:29:58.177 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:58.177+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:58.236 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:58.233+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:58.264 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:58.264 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:58.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:29:58.365 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:58.365+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:58.438 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:58.436+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:58.444 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:58.444 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:58.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:29:58.545 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:58.545+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:58.623 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:58.623 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:58.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:29:58.640 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:58.640+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:58.741 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:58.741+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:58.765 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:29:58.814 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:58.814 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:58.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-06T00:29:58.841 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:58.840+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:58.943 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:29:58.940+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:29:59.002 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:59.002 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:59.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-06T00:29:59.065 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:59.065+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:59.138 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:59.138 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:59.144 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:59.141+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:59.152 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:29:59.245 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:59.245+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:59.322 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:59.322 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:59.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:29:59.346 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:29:59.344+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:29:59.446 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:29:59.445+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:29:59.502 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:59.502 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:59.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:29:59.546 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:29:59.544+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:29:59.647 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:29:59.648 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:59.645+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:59.686 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:29:59.686 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:29:59.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:29:59.748 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:59.749+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:59.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:29:59.815 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:29:59.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:29:59.834 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:29:59.838 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:59.838 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:29:59.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:27136 3.18 3:1e04d4de:::rbd_header.110ce6f30959:head [call rbd.get_size in=19b out=9b,call rbd.get_features in=24b out=16b,call rbd.get_flags in=20b out=8b,call rbd.get_snapcontext in=18b,call lock.get_info in=30b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e174)", 2024-05-06T00:29:59.838 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:29:59.840525+0000", 2024-05-06T00:29:59.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.00025995899999999998, 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029341000000000002, 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 27136 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:59.840525+0000", 2024-05-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:59.840525+0000", 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:59.840526+0000", 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2399999999999998e-07 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:59.840529+0000", 2024-05-06T00:29:59.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1480000000000002e-06 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:59.840536+0000", 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5850000000000002e-06 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:59.840543+0000", 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9469999999999997e-06 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:59.840578+0000", 2024-05-06T00:29:59.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4570000000000003e-05 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:29:59.840624+0000", 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6588999999999997e-05 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 1 2024-05-06T00:29:59.842 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:59.850 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:29:59.849+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:29:59.853 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:29:59.853 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:29:59.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:29:59.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:29:59.874 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:29:59.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-06T00:29:59.896 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-06T00:29:59.919 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:29:59.919 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:29:59.950 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:29:59.949+0000 7f94ca9a1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 24055) UID: 0 2024-05-06T00:29:59.975 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:29:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:29:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:29:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:29:59.975 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:29:59.989 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:30:00.076 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:00.076+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:00.153 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:00.153+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:00.157 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 181.352132066, 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:30:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:30:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:30:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:30:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 181.35211586299999, 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:30:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:30:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:30:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343885646, 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343854276, 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:30:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343820487, 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:30:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.34364003100001, 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:30:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:30:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:30:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:30:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:30:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:30:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343606388, 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343582606, 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:30:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.34355632899999, 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:30:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:30:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.34351308799999, 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:30:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:00.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:30:00.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:30:00.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343483459, 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:30:00.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:30:00.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343450609, 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:30:00.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:30:00.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:30:00.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:30:00.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:30:00.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.231 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.34341897100001, 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:30:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343386169, 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:30:00.237 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:30:00.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:30:00.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:30:00.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:30:00.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:30:00.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:30:00.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:30:00.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343354136, 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:30:00.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.34332205699999, 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 115.343290749, 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:30:00.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:30:00.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.503145032000006, 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:30:00.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:30:00.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:30:00.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.501738883000002, 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:30:00.269 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:00.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:00.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:30:00.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.276 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.501685925000004, 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.277 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:30:00.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.283 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:00.285 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:30:00.286 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:00.285+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:00.386 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:00.384+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:00.458 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:00.458 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:00.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:30:00.486 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:00.485+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:00.516 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-05-06T00:30:00.516 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:30:00.516 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5] 2024-05-06T00:30:00.516 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-06T00:30:00.516 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:30:00.587 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:00.584+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:00.597 INFO:tasks.ceph.osd.4:Stopped 2024-05-06T00:30:00.597 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-05-06T00:30:00.597 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:30:00.597 DEBUG:teuthology.orchestra.run.smithi146:> type ceph-objectstore-tool 2024-05-06T00:30:00.602 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-05-06T00:30:00.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-05-06T00:30:00.640 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:00.640 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:00.640 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:00.641 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:00.641 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:00.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:30:00.696 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:00.696+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:00.789 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:00.789+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:00.793 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:30:00.826 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:00.826 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:00.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.800285448, 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:30:00.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:30:00.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.80026465900001, 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.800244597, 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.836 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.840 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.800225321, 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:30:00.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.800205203, 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.846 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:00.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.800173794, 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:30:00.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.80014088199999, 2024-05-06T00:30:00.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:30:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:30:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:30:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:30:00.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.800120295, 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:00.864 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.800100915, 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:00.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.800056234, 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.869 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.799991143, 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:30:00.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.877 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.79995956899999, 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.799937308, 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.799916192, 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:30:00.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:30:00.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:30:00.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.142199566, 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.891 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:30:00.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 116.141789324, 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:30:00.895 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.899 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.203144730000005, 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:30:00.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:30:00.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.203073446000005, 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:00.905 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:30:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:30:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.203041812999999, 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.198823266999995, 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:00.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:30:00.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:30:00.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:30:00.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:00.922 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:00.922 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.e 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.11 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.a 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.14 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.1b 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.1 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.1a 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.1c 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.1d 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.19 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.3 2024-05-06T00:30:00.923 INFO:teuthology.orchestra.run.smithi146.stdout:3.9 2024-05-06T00:30:00.924 INFO:teuthology.orchestra.run.smithi146.stdout:2.7 2024-05-06T00:30:00.924 INFO:teuthology.orchestra.run.smithi146.stdout:3.15 2024-05-06T00:30:00.924 INFO:teuthology.orchestra.run.smithi146.stdout:3.6 2024-05-06T00:30:00.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:30:00.925 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:00.920+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:01.024 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:01.024+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:01.089 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:01.089 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:01.104 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:30:01.148 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:01.148+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:01.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 3.a --file /var/log/ceph/exp.3.a.4 2024-05-06T00:30:01.225 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:01.224+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:01.269 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:01.269 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:01.269 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:01.269 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:01.269 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:01.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:30:01.326 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:01.325+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:01.427 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:01.425+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.44364434600001, 2024-05-06T00:30:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:01.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:30:01.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.455 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.44341577399999, 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:30:01.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:30:01.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.44333849, 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:30:01.460 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:30:01.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:30:01.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.44314250900001, 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:30:01.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.536033133, 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:30:01.469 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:01.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.535972254, 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:30:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:30:01.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.41841580400001, 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.481 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:01.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:01.485 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.418371076, 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.489 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.418332675, 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:30:01.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:30:01.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.418295413, 2024-05-06T00:30:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:30:01.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:30:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.41825850399999, 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.499 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:30:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:30:01.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.418220586, 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:30:01.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.41818347100001, 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:30:01.507 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:30:01.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.41240302200001, 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:30:01.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:30:01.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:30:01.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.412364938, 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:01.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.412327264, 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:30:01.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:30:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.402891754, 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.528 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.402859327, 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:30:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:30:01.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.40283544499999, 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.402815515, 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:01.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:01.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:01.541 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:01.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Exporting 3.a info 3.a( v 174'747 (0'0,174'747] local-lis/les=107/108 n=52 ec=18/18 lis/c=107/107 les/c/f=108/108/0 sis=107) 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:5016d400:::rbd_data.110cd5ddb45c.0000000000000128:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:5029607f:::rbd_data.110cc8beeb04.0000000000000017:6# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:5029607f:::rbd_data.110cc8beeb04.0000000000000017:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:502d5df8:::rbd_data.110cd5ddb45c.0000000000000081:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:50645f78:::rbd_data.110c915b8194.000000000000002d:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:5074298e:::rbd_data.110cd5ddb45c.0000000000000117:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:508bb5c6:::rbd_data.110cf57807c6.00000000000000ae:d# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:508bb5c6:::rbd_data.110cf57807c6.00000000000000ae:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:50c17ca4:::rbd_data.110cd5ddb45c.0000000000000107:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:50dd91af:::rbd_data.110cc6eb3028.000000000000000a:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:50edc290:::rbd_data.110c936bb298.00000000000000e6:head# 2024-05-06T00:30:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:51339c8c:::rbd_data.110cfb96f133.0000000000000015:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:513561cd:::rbd_data.110cd5ddb45c.000000000000008f:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:51765f29:::rbd_data.110cd5ddb45c.00000000000000fa:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:518b5d79:::rbd_data.110cc6eb3028.0000000000000006:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:51a1f983:::rbd_data.110cd5ddb45c.000000000000011c:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:51e7aee5:::rbd_data.110cfb96f133.0000000000000026:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:51f78603:::rbd_object_map.110cd5ddb45c.000000000000000e:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:52426c0c:::rbd_data.110cfb96f133.0000000000000074:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:527252be:::rbd_data.110c287826c9.0000000000000003:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:52befec0:::rbd_data.110cd5ddb45c.00000000000000f7:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:52d2b813:::rbd_data.110cf57807c6.0000000000000000:d# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:52d2b813:::rbd_data.110cf57807c6.0000000000000000:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:52fe87f5:::rbd_data.110cd5ddb45c.0000000000000124:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:535385f0:::rbd_children:head# 2024-05-06T00:30:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:537bc742:::rbd_data.110c936bb298.0000000000000103:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:53802c53:::rbd_data.110cfb96f133.0000000000000025:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:53cd4588:::rbd_data.110c6c438b03.0000000000000006:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:53d1a741:::rbd_data.110c936bb298.00000000000000ea:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:54020152:::rbd_data.110cfb96f133.0000000000000071:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:54041356:::rbd_data.110c936bb298.00000000000000b2:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:542123e8:::rbd_data.110c17d674e9.0000000000000009:4# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:542123e8:::rbd_data.110c17d674e9.0000000000000009:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:54513e33:::rbd_data.110cd5ddb45c.0000000000000102:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:545786ae:::rbd_id.image_client.0-clone13:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:555cbf7c:::rbd_data.110cfb96f133.0000000000000075:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:55854a91:::rbd_data.110c936bb298.00000000000000d4:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:55b72936:::rbd_data.110cf57807c6.000000000000002d:d# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:55b72936:::rbd_data.110cf57807c6.000000000000002d:head# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10# 2024-05-06T00:30:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:55da1466:::rbd_data.110c5ae73073.0000000000000002:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:560873e5:::rbd_data.110c6af4f54c.0000000000000008:a# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:560873e5:::rbd_data.110c6af4f54c.0000000000000008:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:567d396c:::rbd_data.110c936bb298.0000000000000090:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:568ad3cc:::rbd_data.110cd55b09e5.0000000000000002:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:56a5f987:::rbd_data.110cf57807c6.000000000000002b:d# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:56a5f987:::rbd_data.110cf57807c6.000000000000002b:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:56a80790:::rbd_data.110cd5ddb45c.000000000000012a:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:56a9256d:::rbd_data.110cd5ddb45c.0000000000000022:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:56be37f1:::rbd_data.110cd5ddb45c.000000000000005b:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:5739b187:::rbd_object_map.110c7776112a.0000000000000008:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Read #3:57bf73b9:::rbd_header.110cd55b09e5:head# 2024-05-06T00:30:01.552 INFO:teuthology.orchestra.run.smithi146.stderr:Export successful 2024-05-06T00:30:01.553 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:01.541+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:01.625 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:30:01.643 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:01.640+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:01.682 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:30:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:27269 3.11 3:8c17c874:::rbd_object_map.110c5c796250:head [call rbd.object_map_load in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e176)", 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:30:01.683483+0000", 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.000213668, 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026159700000000002, 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 27269 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.683483+0000", 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.683483+0000", 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.683485+0000", 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0770000000000002e-06 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.683496+0000", 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0953e-05 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.683528+0000", 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2366999999999997e-05 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.683553+0000", 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5006000000000001e-05 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.683631+0000", 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8182999999999993e-05 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.683687+0000", 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5865000000000001e-05 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-05-06T00:30:01.686 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:01.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:30:01.744 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:01.744+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:01.820 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:01.820 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:01.820 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:30:01.820 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:30:01.820 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:01.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:30:01.845 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:01.844+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:01.852 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 3.a 2024-05-06T00:30:01.940 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 212.356143143, 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 164.083370589, 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:30:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:30:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:30:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 164.08326864599999, 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:30:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:30:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 164.08315954400001, 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:30:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:30:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:30:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 164.08306168499999, 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:30:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 164.082996635, 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:30:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:30:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:30:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 164.08290398899999, 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:30:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 164.08283198500001, 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:30:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.32337049899999, 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.32335125099999, 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:30:02.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:30:02.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:02.019 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.32330113099999, 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:02.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:30:02.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.32328007999999, 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.025 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:30:02.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:30:02.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.32326476099999, 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.031 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:30:02.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.32324787900001, 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.037 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:30:02.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.323229543, 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:30:02.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.323213541, 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.049 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.054 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 163.32312717799999, 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:30:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [read 12582912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:02.060 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.143876931999998, 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857773900000002, 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.812763+0000", 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99789039000000002 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813015+0000", 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025184099999999999 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813243+0000", 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022833400000000001 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [sparse-read 12587008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:30:02.065 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.143776459999998, 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99922042799999999, 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:02.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813844+0000", 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99888496199999999 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813901+0000", 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6561000000000003e-05 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813986+0000", 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5643999999999995e-05 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.070 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:27247 3.11 3:8c17c874:::rbd_object_map.110c5c796250:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.159091458, 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.84328285300000005, 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 27247 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799453+0000", 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0980000000000001e-06 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799462+0000", 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8859999999999993e-06 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799481+0000", 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9239e-05 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799503+0000", 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2186000000000001e-05 2024-05-06T00:30:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799607+0000", 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103436 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799621+0000", 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.641323+0000", 2024-05-06T00:30:02.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.84170262200000001 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.641474+0000", 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015097 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642589+0000", 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011146859999999999 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:02.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642646+0000", 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6965000000000001e-05 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642734+0000", 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7828e-05 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:02.076 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:02.077 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:01.988+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:02.078 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:02.076+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:02.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:30:02.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:02.097 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:02.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:02.113 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:02.173 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:02.203 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:02.203 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:02.216 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:02.216+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:02.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:30:02.279 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:02.276+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:02.333 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:02.334 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:02.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:30:02.393 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:02.393+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:02.471 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:02.471 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:02.481 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:02.481+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:02.488 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:30:02.582 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:02.580+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:02.620 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:02.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:30:02.683 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:02.681+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:02.785 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:02.781+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:02.808 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:02.808 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:02.823 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:30:02.885 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:02.885+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:02.950 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:02.950 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:02.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:30:02.986 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:02.985+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:03.085 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:03.085 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:03.088 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:03.084+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:03.098 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:30:03.188 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:03.185+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:03.260 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:03.260 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:03.274 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:30:03.320 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:03.320+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:03.390 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:03.388+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:03.406 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:30:03.473 INFO:teuthology.orchestra.run.smithi146.stdout: marking collection for removal 2024-05-06T00:30:03.473 INFO:teuthology.orchestra.run.smithi146.stdout:setting '_remove' omap key 2024-05-06T00:30:03.473 INFO:teuthology.orchestra.run.smithi146.stdout:finish_remove_pgs 3.a_head removing 3.a 2024-05-06T00:30:03.478 INFO:teuthology.orchestra.run.smithi146.stdout:Remove successful 2024-05-06T00:30:03.490 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:03.489+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:03.571 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:03.571 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:03.585 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:30:03.632 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:03.632+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:03.656 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:30:03.693 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:03.693+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:03.704 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:03.704 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:03.719 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:30:03.795 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:03.792+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:03.846 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:03.847 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:03.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:30:03.908 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:03.908+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:03.992 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:03.993 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:03.996 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:03.996+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:04.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:30:04.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.a.4 2024-05-06T00:30:04.116 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:04.117+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:04.135 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:04.135 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:04.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:30:04.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:30:04.237 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:04.237+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:04.298 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:04.296+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:04.312 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:04.312 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:04.325 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:04.326 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:04.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:30:04.342 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:04.343 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:04.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:04.378 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:04.378 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:04.378 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:04.399 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:04.396+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:04.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:04.490 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:04.490 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:04.490 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:30:04.490 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:04.500 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:04.500+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:04.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:30:04.601 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:04.600+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:04.630 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:30:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:30:04.631 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:04.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:30:04.702 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:04.700+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:04.766 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:04.766 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:30:04.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:30:04.766 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:30:04.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.766 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:30:04.766 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:04.766 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.80323866099999, 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:04.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:04.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:30:04.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:30:04.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:30:04.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:30:04.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:30:04.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:30:04.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:04.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:30:04.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:30:04.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:30:04.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:30:04.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:30:04.774 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:04.775 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.80205404599999, 2024-05-06T00:30:04.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:30:04.775 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:30:04.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:04.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:30:04.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.918633001000003, 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.780 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:30:04.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.918595787999998, 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.785 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:30:04.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.918506981999997, 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:30:04.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:30:04.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:30:04.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:30:04.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.796 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.797 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.797 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:04.797 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.918446789999997, 2024-05-06T00:30:04.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:30:04.798 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.798 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.798 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.799 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:30:04.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.799 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:30:04.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.803 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.918392486999998, 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.804 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:30:04.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.918325846999998, 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:30:04.808 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:30:04.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:30:04.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.918263574999997, 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:30:04.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.918204578999998, 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:30:04.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.911799858000002, 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.823 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:30:04.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.911751123999998, 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:30:04.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:30:04.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.911706872000003, 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:30:04.832 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:30:04.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:30:04.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.838 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.838 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:04.838 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.911635840000002, 2024-05-06T00:30:04.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:30:04.838 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:30:04.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:30:04.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:30:04.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:30:04.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:30:04.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:30:04.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.844 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.911585025000001, 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.845 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.846 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:30:04.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.846 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:30:04.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:30:04.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:30:04.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.850 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.850 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.850 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:04.850 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.902845876999997, 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:30:04.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.902822862999997, 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:04.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:30:04.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:30:04.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:30:04.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:30:04.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:30:04.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.902800612999997, 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.862 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:30:04.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.862 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:04.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:04.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:04.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:30:04.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:30:04.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:30:04.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:04.868 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.902690763000003, 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:30:04.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:30:04.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.902626712999997, 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:04.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:04.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:30:04.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:04.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:04.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:04.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:30:04.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:30:04.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.878 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:04.879 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:04.879 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:30:04.881 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:04.877+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:04.881 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:30:04.981 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:04.981+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:05.063 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:05.063 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:05.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:30:05.082 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:05.081+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:05.184 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:05.184+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:05.195 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:05.195 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:05.195 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:05.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:30:05.255 INFO:teuthology.orchestra.run.smithi146.stdout:get_pg_num_history pg_num_history pg_num_history(e174 pg_nums {1={12=1},2={15=8},3={18=32}} deleted_pools ) 2024-05-06T00:30:05.255 INFO:teuthology.orchestra.run.smithi146.stdout:Importing pgid 3.a 2024-05-06T00:30:05.256 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:5016d400:::rbd_data.110cd5ddb45c.0000000000000128:head# 2024-05-06T00:30:05.256 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.257 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:5029607f:::rbd_data.110cc8beeb04.0000000000000017:6# 2024-05-06T00:30:05.259 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:5029607f:::rbd_data.110cc8beeb04.0000000000000017:head# 2024-05-06T00:30:05.259 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 6=[]:{6=[6]} 2024-05-06T00:30:05.259 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:502d5df8:::rbd_data.110cd5ddb45c.0000000000000081:head# 2024-05-06T00:30:05.260 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.260 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:50645f78:::rbd_data.110c915b8194.000000000000002d:head# 2024-05-06T00:30:05.260 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.261 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:5074298e:::rbd_data.110cd5ddb45c.0000000000000117:head# 2024-05-06T00:30:05.261 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.261 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:508bb5c6:::rbd_data.110cf57807c6.00000000000000ae:d# 2024-05-06T00:30:05.263 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:508bb5c6:::rbd_data.110cf57807c6.00000000000000ae:head# 2024-05-06T00:30:05.263 INFO:teuthology.orchestra.run.smithi146.stdout:snapset d=[]:{d=[d]} 2024-05-06T00:30:05.264 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:50c17ca4:::rbd_data.110cd5ddb45c.0000000000000107:head# 2024-05-06T00:30:05.264 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.264 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:50dd91af:::rbd_data.110cc6eb3028.000000000000000a:head# 2024-05-06T00:30:05.264 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.264 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:50edc290:::rbd_data.110c936bb298.00000000000000e6:head# 2024-05-06T00:30:05.265 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 2b=[]:{} 2024-05-06T00:30:05.265 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:51339c8c:::rbd_data.110cfb96f133.0000000000000015:head# 2024-05-06T00:30:05.265 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 1a=[]:{} 2024-05-06T00:30:05.265 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:513561cd:::rbd_data.110cd5ddb45c.000000000000008f:head# 2024-05-06T00:30:05.266 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.266 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:51765f29:::rbd_data.110cd5ddb45c.00000000000000fa:head# 2024-05-06T00:30:05.266 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.266 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:518b5d79:::rbd_data.110cc6eb3028.0000000000000006:head# 2024-05-06T00:30:05.266 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.267 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:51a1f983:::rbd_data.110cd5ddb45c.000000000000011c:head# 2024-05-06T00:30:05.267 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.267 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:51e7aee5:::rbd_data.110cfb96f133.0000000000000026:head# 2024-05-06T00:30:05.267 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 1a=[]:{} 2024-05-06T00:30:05.267 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:51f78603:::rbd_object_map.110cd5ddb45c.000000000000000e:head# 2024-05-06T00:30:05.268 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.268 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:52426c0c:::rbd_data.110cfb96f133.0000000000000074:head# 2024-05-06T00:30:05.268 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 1a=[]:{} 2024-05-06T00:30:05.268 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:527252be:::rbd_data.110c287826c9.0000000000000003:head# 2024-05-06T00:30:05.269 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.270 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:52befec0:::rbd_data.110cd5ddb45c.00000000000000f7:head# 2024-05-06T00:30:05.270 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.270 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:52d2b813:::rbd_data.110cf57807c6.0000000000000000:d# 2024-05-06T00:30:05.271 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:52d2b813:::rbd_data.110cf57807c6.0000000000000000:head# 2024-05-06T00:30:05.271 INFO:teuthology.orchestra.run.smithi146.stdout:snapset d=[]:{d=[d]} 2024-05-06T00:30:05.272 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:52fe87f5:::rbd_data.110cd5ddb45c.0000000000000124:head# 2024-05-06T00:30:05.272 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.272 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:535385f0:::rbd_children:head# 2024-05-06T00:30:05.272 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.273 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:537bc742:::rbd_data.110c936bb298.0000000000000103:head# 2024-05-06T00:30:05.273 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 2b=[]:{} 2024-05-06T00:30:05.273 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:53802c53:::rbd_data.110cfb96f133.0000000000000025:head# 2024-05-06T00:30:05.273 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 1a=[]:{} 2024-05-06T00:30:05.274 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:53cd4588:::rbd_data.110c6c438b03.0000000000000006:head# 2024-05-06T00:30:05.274 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.277 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:53d1a741:::rbd_data.110c936bb298.00000000000000ea:head# 2024-05-06T00:30:05.277 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 2b=[]:{} 2024-05-06T00:30:05.277 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:54020152:::rbd_data.110cfb96f133.0000000000000071:head# 2024-05-06T00:30:05.278 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 1a=[]:{} 2024-05-06T00:30:05.278 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:54041356:::rbd_data.110c936bb298.00000000000000b2:head# 2024-05-06T00:30:05.278 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 2b=[]:{} 2024-05-06T00:30:05.278 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:542123e8:::rbd_data.110c17d674e9.0000000000000009:4# 2024-05-06T00:30:05.287 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:542123e8:::rbd_data.110c17d674e9.0000000000000009:head# 2024-05-06T00:30:05.287 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 4=[]:{4=[4]} 2024-05-06T00:30:05.288 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:54513e33:::rbd_data.110cd5ddb45c.0000000000000102:head# 2024-05-06T00:30:05.288 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.288 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:545786ae:::rbd_id.image_client.0-clone13:head# 2024-05-06T00:30:05.288 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.289 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:555cbf7c:::rbd_data.110cfb96f133.0000000000000075:head# 2024-05-06T00:30:05.289 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 1a=[]:{} 2024-05-06T00:30:05.289 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:55854a91:::rbd_data.110c936bb298.00000000000000d4:head# 2024-05-06T00:30:05.289 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 2b=[]:{} 2024-05-06T00:30:05.290 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:55b72936:::rbd_data.110cf57807c6.000000000000002d:d# 2024-05-06T00:30:05.291 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:55b72936:::rbd_data.110cf57807c6.000000000000002d:head# 2024-05-06T00:30:05.291 INFO:teuthology.orchestra.run.smithi146.stdout:snapset d=[]:{d=[d]} 2024-05-06T00:30:05.291 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:55da1466:::rbd_data.110c5ae73073.0000000000000002:10# 2024-05-06T00:30:05.296 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:05.296+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:05.305 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:55da1466:::rbd_data.110c5ae73073.0000000000000002:head# 2024-05-06T00:30:05.307 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 10=[]:{10=[10]} 2024-05-06T00:30:05.311 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:560873e5:::rbd_data.110c6af4f54c.0000000000000008:a# 2024-05-06T00:30:05.312 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:560873e5:::rbd_data.110c6af4f54c.0000000000000008:head# 2024-05-06T00:30:05.313 INFO:teuthology.orchestra.run.smithi146.stdout:snapset a=[]:{a=[a]} 2024-05-06T00:30:05.313 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:567d396c:::rbd_data.110c936bb298.0000000000000090:head# 2024-05-06T00:30:05.313 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 2b=[]:{} 2024-05-06T00:30:05.313 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:568ad3cc:::rbd_data.110cd55b09e5.0000000000000002:head# 2024-05-06T00:30:05.314 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.317 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:56a5f987:::rbd_data.110cf57807c6.000000000000002b:d# 2024-05-06T00:30:05.318 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:56a5f987:::rbd_data.110cf57807c6.000000000000002b:head# 2024-05-06T00:30:05.318 INFO:teuthology.orchestra.run.smithi146.stdout:snapset d=[]:{d=[d]} 2024-05-06T00:30:05.319 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:56a80790:::rbd_data.110cd5ddb45c.000000000000012a:head# 2024-05-06T00:30:05.319 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.319 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:56a9256d:::rbd_data.110cd5ddb45c.0000000000000022:head# 2024-05-06T00:30:05.319 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.320 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:56be37f1:::rbd_data.110cd5ddb45c.000000000000005b:head# 2024-05-06T00:30:05.320 INFO:teuthology.orchestra.run.smithi146.stdout:snapset e=[]:{} 2024-05-06T00:30:05.321 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:5739b187:::rbd_object_map.110c7776112a.0000000000000008:head# 2024-05-06T00:30:05.321 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.321 INFO:teuthology.orchestra.run.smithi146.stdout:Write #3:57bf73b9:::rbd_header.110cd55b09e5:head# 2024-05-06T00:30:05.322 INFO:teuthology.orchestra.run.smithi146.stdout:snapset 0=[]:{} 2024-05-06T00:30:05.322 INFO:teuthology.orchestra.run.smithi146.stdout:write_pg epoch 174 info 3.a( v 174'747 (0'0,174'747] local-lis/les=107/108 n=52 ec=18/18 lis/c=107/107 les/c/f=108/108/0 sis=107) 2024-05-06T00:30:05.325 INFO:teuthology.orchestra.run.smithi146.stdout:Import successful 2024-05-06T00:30:05.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:30:05.381 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 186.57541432400001, 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:30:05.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.387 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 186.57539812100001, 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.393 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.567167904, 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.398 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.567136534, 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.567102745, 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:30:05.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566922289, 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:30:05.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:30:05.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566888646, 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.414 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:30:05.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566864864, 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:30:05.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:30:05.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.56683858700001, 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:30:05.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:30:05.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:30:05.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.56679534600001, 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:30:05.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:30:05.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:30:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566765717, 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:30:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:30:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566732867, 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:30:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:30:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566701229, 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:30:05.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:30:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:30:05.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566668427, 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:30:05.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:30:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566636394, 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:30:05.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:30:05.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.56660431500001, 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:30:05.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:30:05.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 120.566573007, 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:05.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:30:05.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:05.471 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.726427290000004, 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:05.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.725021140999999, 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.477 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:05.483 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.724968183000001, 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:05.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:30:05.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:05.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:05.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:30:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:30:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:05.489 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:05.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:05.491 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:30:05.491 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:05.488+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:05.591 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:05.588+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:05.660 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:05.660 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:05.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:30:05.716 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:05.716+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:05.793 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:05.792+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:05.850 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:05.851 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:05.851 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:05.866 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:30:05.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /var/log/ceph/exp.3.a.4 2024-05-06T00:30:05.924 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:05.925+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:05.994 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:05.992+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:06.048 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:06.048 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020408199, 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:06.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:30:06.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:30:06.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.02038741, 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:30:06.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.02036734799999, 2024-05-06T00:30:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:30:06.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:30:06.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020348072, 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:30:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020327954, 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.070 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.02029654499999, 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.074 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020263633, 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:30:06.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020243046, 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:30:06.079 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:30:06.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:30:06.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:30:06.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.02022366600001, 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:30:06.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:06.088 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020178985, 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:06.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020113894, 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.02008232, 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020060059, 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:06.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.106 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 122.020038943, 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:30:06.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.36232231699999, 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.36191207500001, 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:06.119 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.423267480999996, 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:30:06.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:30:06.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.423196196999996, 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:30:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:30:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:30:06.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.423164564000004, 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.130 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:30:06.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:30:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.418946018, 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:30:06.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:30:06.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:30:06.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:30:06.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:06.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:06.143 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:30:06.143 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:06.140+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:06.243 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:06.240+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:06.310 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:06.310 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:06.310 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:06.310 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:06.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:30:06.368 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:06.368+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:06.404 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:30:06.460 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:06.460+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:06.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:06.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:06.493 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:06.493 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:06.493 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:06.508 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:30:06.547 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:06.545+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:06.607 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:06.607 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:06.607 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:06.629 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:06.630 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:06.634 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:06.647 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:06.645+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:06.674 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:06.674 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:06.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:30:06.747 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:06.744+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:06.825 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:06.825 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:06.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:30:06.848 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:06.848+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:06.882 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:30:06.949 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:06.949+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:06.964 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:06.964 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:06.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:30:07.069 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:07.069+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:07.144 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:07.144 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:07.150 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:07.148+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:07.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:30:07.252 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:07.252+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:07.317 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:07.317 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:07.331 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:30:07.352 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:07.352+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:07.453 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:07.453+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:07.454 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:07.454 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:07.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:30:07.555 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:07.552+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:07.634 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:07.634 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:07.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:30:07.655 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:07.653+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:07.756 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:07.757+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:07.795 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:30:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:30:07.896 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:07.896+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:07.924 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:07.924 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:07.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:30:07.957 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:07.957+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:08.057 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:08.057+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:08.100 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:08.100 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:08.114 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:30:08.158 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:08.157+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:08.231 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:08.231 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:08.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:30:08.288 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:08.288+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:08.360 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:08.357+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:08.364 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:08.364 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:08.378 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:30:08.461 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:08.461+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:08.536 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:08.537 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:08.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:30:08.596 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:08.596+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:08.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:30:08.662 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:08.661+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:08.668 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:08.668 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:08.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:30:08.776 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:08.776+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:08.785 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:08.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:08.802 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:08.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:08.819 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:08.846 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:08.846 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:08.847 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:08.849 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:08.849 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:08.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:30:08.864 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:08.860+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:08.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:08.867 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:08.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:08.964 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:08.964+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 160.97291557200001, 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:30:08.979 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:08.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:08.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 160.97268700000001, 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:30:08.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:30:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:08.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:08.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 160.97260971599999, 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:08.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:30:08.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 160.972413735, 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:08.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:08.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:08.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:08.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 128.06530435900001, 2024-05-06T00:30:08.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:30:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:30:08.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.000 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 128.06524347999999, 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.94768703, 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:30:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:30:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:30:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.94764230200001, 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.94760390099999, 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.013 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.947566639, 2024-05-06T00:30:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:30:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.94752973, 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.947491812, 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:30:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.947454697, 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:30:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:30:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.941674248, 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:30:09.034 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:09.038 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.941636164, 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.94159849, 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:30:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.93216298, 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:30:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:30:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.93213055299999, 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.054 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.932106671, 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:09.058 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 124.93208674100001, 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:30:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:09.062 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:09.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:30:09.108 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:09.108+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:09.166 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:09.165+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:09.199 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:30:09.200 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:09.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:30:09.266 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:09.264+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:09.367 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:09.364+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:09.375 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:09.375 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:09.375 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:30:09.375 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:30:09.375 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:09.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:30:09.469 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:09.468+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 219.91291208499999, 2024-05-06T00:30:09.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:30:09.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:30:09.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:30:09.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 171.64013953099999, 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.525 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:30:09.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:30:09.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 171.64003758800001, 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:30:09.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:30:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 171.639928486, 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:30:09.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.542 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 171.63983062700001, 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:30:09.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:30:09.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 171.63976557699999, 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.548 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:30:09.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:30:09.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:30:09.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:30:09.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:30:09.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:30:09.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:30:09.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:30:09.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:30:09.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:30:09.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:30:09.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 171.63967293100001, 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:09.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:30:09.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 171.639600927, 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.564 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.88013944100001, 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.570 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.571 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:30:09.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.571 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:09.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:09.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:09.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:30:09.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:30:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.88012019300001, 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:30:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:30:09.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.88007007300001, 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:30:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:30:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.88004902200001, 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.590 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:30:09.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:30:09.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.88003370300001, 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.596 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:30:09.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.880016821, 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:30:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:30:09.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:30:09.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.87999848499999, 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:30:09.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:30:09.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:30:09.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:30:09.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.87998248299999, 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:09.618 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 170.87989612000001, 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:09.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:30:09.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:30:09.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [read 12582912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.700645874000003, 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857773900000002, 2024-05-06T00:30:09.624 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.812763+0000", 2024-05-06T00:30:09.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99789039000000002 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813015+0000", 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025184099999999999 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813243+0000", 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022833400000000001 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [sparse-read 12587008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.700545402000003, 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99922042799999999, 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:30:09.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:30:09.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813844+0000", 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99888496199999999 2024-05-06T00:30:09.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813901+0000", 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6561000000000003e-05 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813986+0000", 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5643999999999995e-05 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:27247 3.11 3:8c17c874:::rbd_object_map.110c5c796250:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7158604000000004, 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.84328285300000005, 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 27247 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:09.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799453+0000", 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0980000000000001e-06 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:09.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799462+0000", 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8859999999999993e-06 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799481+0000", 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9239e-05 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799503+0000", 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2186000000000001e-05 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799607+0000", 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103436 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799621+0000", 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.641323+0000", 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.84170262200000001 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:09.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.641474+0000", 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015097 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642589+0000", 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011146859999999999 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642646+0000", 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6965000000000001e-05 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:09.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642734+0000", 2024-05-06T00:30:09.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7828e-05 2024-05-06T00:30:09.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.640 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:09.640 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:09.640 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:09.641 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:30:09.642 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:09.640+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:09.742 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:09.744+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:09.767 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:30:09.767 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:09.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:30:09.845 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:09.841+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:09.853 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-06T00:30:09.911 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:30:09.911 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:30:09.911 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:09.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:30:09.973 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:09.973+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:10.045 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:10.044+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:10.094 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.13216044399999, 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:30:10.095 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:30:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.13097582899999, 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:30:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:30:10.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:10.105 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.247554784000002, 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.109 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.247517571000003, 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:30:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:30:10.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.247428765000002, 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:30:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.247368573000003, 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:30:10.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.247314269999997, 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:30:10.123 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:30:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.247247629999997, 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:30:10.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.132 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.247185358000003, 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:30:10.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:30:10.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:30:10.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.247126362000003, 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.141 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.240721641, 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:30:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.240672906999997, 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:30:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:30:10.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:30:10.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.240628655000002, 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:30:10.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:30:10.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.240557623000001, 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.155 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:30:10.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.240506807999999, 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:30:10.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.231767660000003, 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.164 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:30:10.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:30:10.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.231744646000003, 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:30:10.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.231722396000002, 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.173 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.177 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.231612546000001, 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:30:10.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:30:10.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.231548496000002, 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:10.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:30:10.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:30:10.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:10.186 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:10.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:30:10.189 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:10.188+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:10.288 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:10.289+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:10.312 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:10.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:10.313 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:10.313 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:10.326 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:30:10.389 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:10.389+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:10.449 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:10.449 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:10.450 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:10.450 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:10.450 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:10.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:30:10.490 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:10.489+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:10.589 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:10.589 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:10.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:10.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:10.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:30:10.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:10.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 191.78506230100001, 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:30:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 191.78504609800001, 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.596 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:30:10.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776815881, 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:30:10.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776784511, 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:30:10.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.614 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776750722, 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.615 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:30:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:10.620 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.77657026599999, 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:30:10.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776536623, 2024-05-06T00:30:10.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:30:10.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:30:10.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:10.632 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776512841, 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776486564, 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:30:10.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.644 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776443323, 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.645 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:30:10.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776413694, 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.655 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776380844, 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.656 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:30:10.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:10.661 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.77634920600001, 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:30:10.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:30:10.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776316404, 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:30:10.667 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:30:10.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:30:10.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776284371, 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:10.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:30:10.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:30:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:30:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:30:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:30:10.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.776252292, 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.681 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:30:10.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.77622098400001, 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:30:10.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:10.691 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.936075267000007, 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.934669118000002, 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:30:10.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:30:10.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.934616160000004, 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:30:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:30:10.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.711 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:10.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:30:10.713 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:10.712+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:10.813 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:10.813+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:10.842 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-06T00:30:10.872 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:30:10.892 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:28721 3.2 3:46f4a6be:::rbd_header.110cc6eb3028:head [call lock.lock in=76b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:30:10.894326+0000", 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0010228500000000001, 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010450030000000001, 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for sub ops", 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 28721 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.893 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894326+0000", 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894326+0000", 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894327+0000", 2024-05-06T00:30:10.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1319999999999999e-06 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894333+0000", 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8109999999999999e-06 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894350+0000", 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7282000000000002e-05 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894360+0000", 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9820000000000003e-06 2024-05-06T00:30:10.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894404+0000", 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3538999999999998e-05 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894444+0000", 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0652999999999999e-05 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.894536+0000", 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1688999999999996e-05 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:10.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:30:10.895162+0000", 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00062643599999999996 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 1 2024-05-06T00:30:10.897 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:10.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:30:10.914 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:10.913+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:10.944 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5] 2024-05-06T00:30:10.944 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:30:10.944 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-06T00:30:10.944 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-06T00:30:11.015 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:11.013+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:11.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:11.048 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:11.048 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:11.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:11.066 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:11.080 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:11.080 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:11.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:11.089 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:11.089 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:11.101 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:30:11.110 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:11.111 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:11.114 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:11.115 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:11.113+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:11.216 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:11.213+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:11.224 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:11.225 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:11.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 185, 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b583c614-b2ca-4769-90c0-44726bf69b34", 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-06T00:26:16.923985+0000", 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-06T00:30:10.842580+0000", 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-06T00:30:00.771526+0000", 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-06T00:28:54.626879+0000", 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5806080, 2024-05-06T00:30:11.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "nodeep-scrub", 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "noscrub", 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-06T00:30:11.262 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 6, 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "luminous", 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:28.436494+0000", 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:30:11.263 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-06T00:30:11.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "14", 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:30:11.265 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:30:11.266 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:30:11.267 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 5, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 5, 2024-05-06T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.40000000596046448, 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.61445111036300659, 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.74000000953674316, 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.61000001430511475 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:33.941780+0000", 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-06T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-06T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "error_message": "pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.7), read balance score may not be reliable", 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 0, 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 0 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-06T00:26:42.622595+0000", 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-06T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-06T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "185", 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 46, 2024-05-06T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 185, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-06T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-06T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "error_message": "pool 3 has some PGs where all OSDs are with primary_affinity 0 (at least pg 3.a), read balance score may not be reliable", 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 0, 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 0 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d31b9a6a-864b-4c24-99db-8cc6cf3a5c24", 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 175, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-06T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-06T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1427564096 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6817/1427564096", 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6819/1427564096", 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6823/1427564096", 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6821/1427564096", 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "86509826-0e53-4678-8260-17c97c05af54", 2024-05-06T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 175, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-06T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-06T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2663808859 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6803/2663808859", 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6807/2663808859", 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/2663808859", 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6811/2663808859", 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "8973b524-521e-41c0-87f2-85cb390fbfeb", 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.2727203369140625, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 175, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 385938386 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6802/385938386", 2024-05-06T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6806/385938386", 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6814/385938386", 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6810/385938386", 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "3b0d07bd-132e-48ff-83ac-7182eee05df9", 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0, 2024-05-06T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 175, 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6811", 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6812", 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6814", 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6816", 2024-05-06T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6821", 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6822", 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6817", 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6819", 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4178038317 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6812/4178038317", 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6816/4178038317", 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6822/4178038317", 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6819/4178038317", 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2", 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 0, 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.833282470703125, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 132, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 175, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6808", 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6809", 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6810", 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6813", 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6820", 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6823", 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6815", 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6818", 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1052555350 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6809/1052555350", 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6813/1052555350", 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1052555350", 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6818/1052555350", 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "exists" 2024-05-06T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "34b61f5f-e407-4414-8bc5-11263bbe261a", 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.5807037353515625, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 11, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 175, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-06T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6800", 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6801", 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6802", 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6803", 2024-05-06T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6806", 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6807", 2024-05-06T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6804", 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-06T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6805", 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2019146845 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6801/2019146845", 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.146:6803/2019146845", 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.146:6807/2019146845", 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.146:6805/2019146845", 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.276988+0000", 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.054473+0000", 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.154558+0000", 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.284746+0000", 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.259847+0000", 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 175 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-06T00:26:24.180375+0000", 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-06T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [ 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.15", 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: 4, 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [ 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "2.1", 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 2, 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 3 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.6", 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 1, 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 4 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-06T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [], 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [ 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 12, 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 3 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 16, 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 7 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [ 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-06T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 12, 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 1 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-06T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-06T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-06T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-06T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-06T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:11.323 INFO:tasks.thrashosds.thrasher:j is {'epoch': 185, 'fsid': 'b583c614-b2ca-4769-90c0-44726bf69b34', 'created': '2024-05-06T00:26:16.923985+0000', 'modified': '2024-05-06T00:30:10.842580+0000', 'last_up_change': '2024-05-06T00:30:00.771526+0000', 'last_in_change': '2024-05-06T00:28:54.626879+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T00:26:28.436494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5, 'score_stable': 5, 'optimal_score': 0.4000000059604645, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6144511103630066, 'average_primary_affinity': 0.7400000095367432, 'average_primary_affinity_weighted': 0.6100000143051147}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-05-06T00:26:33.941780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'error_message': 'pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.7), read balance score may not be reliable', 'score_acting': 0, 'score_stable': 0}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-06T00:26:42.622595+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 46, 'snap_epoch': 185, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'error_message': 'pool 3 has some PGs where all OSDs are with primary_affinity 0 (at least pg 3.a), read balance score may not be reliable', 'score_acting': 0, 'score_stable': 0}}], 'osds': [{'osd': 0, 'uuid': 'd31b9a6a-864b-4c24-99db-8cc6cf3a5c24', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 175, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6817', 'nonce': 1427564096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6818', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 1427564096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6822', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 1427564096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1427564096}, {'type': 'v1', 'addr': '172.21.15.78:6821', 'nonce': 1427564096}]}, 'public_addr': '172.21.15.78:6817/1427564096', 'cluster_addr': '172.21.15.78:6819/1427564096', 'heartbeat_back_addr': '172.21.15.78:6823/1427564096', 'heartbeat_front_addr': '172.21.15.78:6821/1427564096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '86509826-0e53-4678-8260-17c97c05af54', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 175, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6801', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6803', 'nonce': 2663808859}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6805', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6807', 'nonce': 2663808859}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6813', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 2663808859}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6809', 'nonce': 2663808859}, {'type': 'v1', 'addr': '172.21.15.78:6811', 'nonce': 2663808859}]}, 'public_addr': '172.21.15.78:6803/2663808859', 'cluster_addr': '172.21.15.78:6807/2663808859', 'heartbeat_back_addr': '172.21.15.78:6815/2663808859', 'heartbeat_front_addr': '172.21.15.78:6811/2663808859', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8973b524-521e-41c0-87f2-85cb390fbfeb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2727203369140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 175, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6802', 'nonce': 385938386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6804', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6806', 'nonce': 385938386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6812', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6814', 'nonce': 385938386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6808', 'nonce': 385938386}, {'type': 'v1', 'addr': '172.21.15.78:6810', 'nonce': 385938386}]}, 'public_addr': '172.21.15.78:6802/385938386', 'cluster_addr': '172.21.15.78:6806/385938386', 'heartbeat_back_addr': '172.21.15.78:6814/385938386', 'heartbeat_front_addr': '172.21.15.78:6810/385938386', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3b0d07bd-132e-48ff-83ac-7182eee05df9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 175, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6811', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6812', 'nonce': 4178038317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6816', 'nonce': 4178038317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6821', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6822', 'nonce': 4178038317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6817', 'nonce': 4178038317}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 4178038317}]}, 'public_addr': '172.21.15.146:6812/4178038317', 'cluster_addr': '172.21.15.146:6816/4178038317', 'heartbeat_back_addr': '172.21.15.146:6822/4178038317', 'heartbeat_front_addr': '172.21.15.146:6819/4178038317', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0.833282470703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 132, 'down_at': 175, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 1052555350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 1052555350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1052555350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6815', 'nonce': 1052555350}, {'type': 'v1', 'addr': '172.21.15.146:6818', 'nonce': 1052555350}]}, 'public_addr': '172.21.15.146:6809/1052555350', 'cluster_addr': '172.21.15.146:6813/1052555350', 'heartbeat_back_addr': '172.21.15.146:6823/1052555350', 'heartbeat_front_addr': '172.21.15.146:6818/1052555350', 'state': ['exists']}, {'osd': 5, 'uuid': '34b61f5f-e407-4414-8bc5-11263bbe261a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5807037353515625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 175, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 2019146845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 2019146845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 2019146845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 2019146845}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 2019146845}]}, 'public_addr': '172.21.15.146:6801/2019146845', 'cluster_addr': '172.21.15.146:6803/2019146845', 'heartbeat_back_addr': '172.21.15.146:6807/2019146845', 'heartbeat_front_addr': '172.21.15.146:6805/2019146845', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.276988+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.054473+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.154558+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.284746+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.259847+0000', 'dead_epoch': 175}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-06T00:26:24.180375+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.15', 'osds': [4, 1]}], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 2, 'to': 3}]}, {'pgid': '3.6', 'mappings': [{'from': 1, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 12, 'length': 3}, {'begin': 16, 'length': 7}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 12, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-06T00:30:11.323 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.15 2024-05-06T00:30:11.323 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap 3.15 2024-05-06T00:30:11.324 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:11.320+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:11.372 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:30:11.464 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:11.464+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:11.524 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:11.524+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:11.544 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:11.544 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:11.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:30:11.644 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:11.644+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:11.720 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:11.720 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:11.726 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:11.725+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:11.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:30:11.827 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:11.825+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stderr:clear 3.15 pg_upmap mapping 2024-05-06T00:30:11.872 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:11.872 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:11.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:30:11.933 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:11.933+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:12.029 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:12.028+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:12.062 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:12.063 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:12.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:30:12.130 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:12.129+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:12.202 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:12.202 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:12.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:30:12.231 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:12.228+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:12.332 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:12.332+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:12.382 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:12.382 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:12.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:30:12.433 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:12.432+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:12.515 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:12.516 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:12.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:30:12.533 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:12.533+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:12.634 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:12.632+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:12.651 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:12.651 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:12.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:30:12.735 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:12.733+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:12.799 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:12.799 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:12.813 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:30:12.860 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:12.860+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:12.870 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:30:12.938 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:12.937+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:12.945 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:12.945 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:12.963 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:30:13.039 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:13.036+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:13.096 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:13.096 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:13.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:30:13.115 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:30:13.141 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:13.141+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:13.242 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:13.240+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:13.275 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:13.275 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:13.288 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:13.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:30:13.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:13.304 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:13.321 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:13.322 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:13.342 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:13.342 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:13.343 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:13.340+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:13.343 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:13.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:13.363 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:13.367 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:13.417 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:13.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:13.417 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:13.417 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:13.417 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:13.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:30:13.443 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:13.441+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:13.544 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:13.541+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:13.545 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51947274400001, 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51945195499999, 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:30:13.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:30:13.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51943189299999, 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.555 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.519412617, 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:30:13.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51939249899999, 2024-05-06T00:30:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:30:13.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:30:13.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51936108999999, 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51932817799999, 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:30:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.519307591, 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.519288211, 2024-05-06T00:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.584 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51924353000001, 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.519178439, 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51914686500001, 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.597 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51912460400001, 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:30:13.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 129.51910348800001, 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:30:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:30:13.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 128.86138686199999, 2024-05-06T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:30:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:30:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 128.86097662, 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:30:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 81.922332026000007, 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 81.922260742000006, 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:30:13.620 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:30:13.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 81.922229109, 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:30:13.625 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 81.918010562999996, 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:30:13.631 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:13.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:30:13.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:30:13.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:13.638 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:13.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:30:13.684 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:13.684+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:13.746 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:13.744+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:13.761 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:13.762 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:13.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:30:13.847 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:13.845+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:13.873 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:30:13.942 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:13.942 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:13.947 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:13.948+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:13.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:30:14.048 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:14.048+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:14.135 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 166.13026202200001, 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:14.136 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:30:14.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 166.13003345000001, 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:30:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:30:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 166.129956166, 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:30:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:30:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 166.12976018500001, 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:14.153 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.22265080899999, 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:30:14.160 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.22258993, 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:14.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:30:14.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.10503348, 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:30:14.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.104988752, 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:14.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:30:14.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.10495035100001, 2024-05-06T00:30:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:30:14.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:30:14.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.10491308900001, 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:30:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:30:14.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.183 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.183 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:14.183 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.10487617999999, 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:30:14.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.10483826199999, 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:30:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:30:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:30:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:30:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:30:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:30:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:30:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:30:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:30:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:30:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:30:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:30:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.10480114699999, 2024-05-06T00:30:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:30:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:30:14.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:30:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.099020698, 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:30:14.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:30:14.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:30:14.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:30:14.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:30:14.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:30:14.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:30:14.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:30:14.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:30:14.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.206 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.09898261399999, 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:30:14.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.09894494, 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.08950942999999, 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.08947700300001, 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:14.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:30:14.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:30:14.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:30:14.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:30:14.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:30:14.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:30:14.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.225 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:14.226 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.08945312099999, 2024-05-06T00:30:14.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:30:14.226 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.226 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:14.226 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.227 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.227 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:30:14.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.227 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:14.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:14.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:30:14.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:30:14.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:30:14.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:30:14.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:30:14.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:30:14.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:30:14.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:30:14.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:30:14.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:30:14.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:30:14.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:30:14.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.234 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.235 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:14.235 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:14.235 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.08943319100001, 2024-05-06T00:30:14.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:30:14.236 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:14.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:14.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:14.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:14.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:14.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:14.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:30:14.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.237 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:14.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:14.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:14.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:14.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:14.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:14.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:14.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:14.241 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:14.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:30:14.243 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:14.241+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:14.344 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:14.341+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:14.374 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:30:14.374 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:14.388 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:30:14.477 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:14.477+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:14.545 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:14.545+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:14.558 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:14.558 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:14.558 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:30:14.558 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:30:14.559 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:14.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:30:14.661 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:14.661+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4174.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 225.138065877, 2024-05-06T00:30:14.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63863410700000001, 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4174", 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/2257567263", 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602399+0000", 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602402+0000", 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7030000000000002e-06 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602416+0000", 2024-05-06T00:30:14.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3696999999999999e-05 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602441+0000", 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5012999999999999e-05 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602461+0000", 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0007e-05 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602614+0000", 2024-05-06T00:30:14.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015314199999999999 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:29.602643+0000", 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9632999999999999e-05 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238054+0000", 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63541054500000005 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238342+0000", 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028818700000000002 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.238793+0000", 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00045081100000000002 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240182+0000", 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013894249999999999 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240864+0000", 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068186500000000003 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.240962+0000", 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7706000000000003e-05 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:26:30.241033+0000", 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1372999999999996e-05 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:596 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.865293323, 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037175046000000003, 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 596 2024-05-06T00:30:14.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875172+0000", 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875173+0000", 2024-05-06T00:30:14.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-06 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875185+0000", 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1905999999999999e-05 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875233+0000", 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7800999999999999e-05 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875250+0000", 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7450000000000001e-05 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907182+0000", 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031931849999999998 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907275+0000", 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2781000000000005e-05 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.907671+0000", 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000396573 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910304+0000", 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026325480000000002 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912175+0000", 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018706630000000001 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912276+0000", 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101378 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912347+0000", 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0785999999999996e-05 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:597 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:14.753 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.86519138, 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038868881000000001, 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:14.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875274+0000", 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875275+0000", 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2729999999999999e-06 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875283+0000", 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9859999999999994e-06 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875302+0000", 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9072999999999998e-05 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875316+0000", 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4396000000000001e-05 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908506+0000", 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033190104999999998 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.908599+0000", 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2523999999999997e-05 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.909126+0000", 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052738600000000002 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913865+0000", 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047381639999999996 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913903+0000", 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8714999999999998e-05 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914034+0000", 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130335 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914142+0000", 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108924 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:598 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.86508227799999, 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044218232000000003, 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.759 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 598 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875383+0000", 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875390+0000", 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.486e-06 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875399+0000", 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875421+0000", 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1673000000000001e-05 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875437+0000", 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6595999999999998e-05 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910363+0000", 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034925375000000002 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.910458+0000", 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4885999999999999e-05 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.911187+0000", 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000728983 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914220+0000", 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030337089999999999 2024-05-06T00:30:14.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919474+0000", 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052538690000000004 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919545+0000", 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0477999999999998e-05 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919601+0000", 2024-05-06T00:30:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6251000000000001e-05 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:601 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.864984419, 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044237385999999997, 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 601 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875481+0000", 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875482+0000", 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.533e-06 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875490+0000", 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6429999999999995e-06 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875510+0000", 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0383000000000001e-05 2024-05-06T00:30:14.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875526+0000", 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512e-05 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912444+0000", 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036917980000000003 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.912575+0000", 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131607 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.913039+0000", 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046328599999999998 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914345+0000", 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013062250000000001 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919618+0000", 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052731610000000002 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919673+0000", 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5254999999999999e-05 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919718+0000", 2024-05-06T00:30:14.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4801e-05 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:604 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.86491936900001, 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044284702000000002, 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 604 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875546+0000", 2024-05-06T00:30:14.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875547+0000", 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2300000000000001e-06 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875554+0000", 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875599+0000", 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4864999999999998e-05 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875615+0000", 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6189e-05 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914649+0000", 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039033483000000001 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.914860+0000", 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021168399999999999 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.915546+0000", 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068537100000000005 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919456+0000", 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039103619999999997 2024-05-06T00:30:14.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919731+0000", 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027505999999999999 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919785+0000", 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3578000000000001e-05 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919830+0000", 2024-05-06T00:30:14.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5683000000000003e-05 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:605 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.86482672299999, 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044948390999999997, 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 605 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875638+0000", 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875640+0000", 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.39e-06 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875647+0000", 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5140000000000004e-06 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875668+0000", 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0355000000000001e-05 2024-05-06T00:30:14.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875681+0000", 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3862999999999999e-05 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916373+0000", 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040691557000000003 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.916591+0000", 2024-05-06T00:30:14.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217789 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.917293+0000", 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00070208399999999995 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.919918+0000", 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026255179999999999 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920378+0000", 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046013500000000003 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920496+0000", 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011741800000000001 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.920587+0000", 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0767999999999996e-05 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.786 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:606 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.86475471899999, 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048613460999999997, 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.787 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875710+0000", 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875712+0000", 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875720+0000", 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5129999999999996e-06 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875759+0000", 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8992000000000002e-05 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.875776+0000", 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6716000000000001e-05 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918023+0000", 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042247390000000003 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918203+0000", 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017930299999999999 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.918854+0000", 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065149499999999998 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924115+0000", 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052608350000000002 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924147+0000", 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1949999999999997e-05 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924249+0000", 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010247 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:17.924324+0000", 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4346999999999998e-05 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.792 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3317 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.105293233, 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03613305, 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3317 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:14.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635172+0000", 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2399999999999999e-07 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635175+0000", 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4779999999999998e-06 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635180+0000", 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2480000000000004e-06 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635183+0000", 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5020000000000001e-06 2024-05-06T00:30:14.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665301+0000", 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030117365 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665438+0000", 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013756900000000001 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.665944+0000", 2024-05-06T00:30:14.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000505591 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668207+0000", 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022627699999999999 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671161+0000", 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029546569999999999 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671243+0000", 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.208e-05 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671305+0000", 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1465999999999997e-05 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3318 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.105273985, 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037448704999999999, 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3318 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635191+0000", 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9200000000000002e-07 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635194+0000", 2024-05-06T00:30:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635198+0000", 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4750000000000004e-06 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635202+0000", 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1110000000000001e-06 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667014+0000", 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031811328 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667094+0000", 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9944999999999994e-05 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.667457+0000", 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000363142 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670156+0000", 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026995109999999999 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672534+0000", 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023778129999999999 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672582+0000", 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8284000000000001e-05 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672640+0000", 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7305000000000001e-05 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3319 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.105223865, 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040645830000000001, 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.804 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635241+0000", 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635242+0000", 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635245+0000", 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0110000000000001e-06 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635251+0000", 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635255+0000", 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9430000000000002e-06 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668287+0000", 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033031234 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668372+0000", 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5131999999999997e-05 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.668811+0000", 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000439283 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672523+0000", 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003711584 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675804+0000", 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032810130000000002 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675852+0000", 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8674000000000002e-05 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675887+0000", 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4598000000000003e-05 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.809 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3320 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.10520281399999, 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040704618999999997, 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.810 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3320 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635262+0000", 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.28e-07 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635265+0000", 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635270+0000", 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0710000000000002e-06 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635273+0000", 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7529999999999999e-06 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669268+0000", 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994374000000001 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669353+0000", 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5150999999999998e-05 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.669672+0000", 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031943 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672715+0000", 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030421269999999999 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675900+0000", 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031855730000000001 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675939+0000", 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8631000000000003e-05 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.675967+0000", 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8036000000000001e-05 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3321 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.105187495, 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041740251999999999, 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3321 2024-05-06T00:30:14.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635277+0000", 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635278+0000", 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3699999999999999e-07 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635280+0000", 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.283e-06 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635284+0000", 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7869999999999999e-06 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635287+0000", 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2150000000000001e-06 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670225+0000", 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034937785999999998 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670333+0000", 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010869 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.670651+0000", 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031778600000000002 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672775+0000", 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002123406 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676895+0000", 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041200500000000001 2024-05-06T00:30:14.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.676963+0000", 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8720000000000006e-05 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677018+0000", 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4292000000000001e-05 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.821 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3322 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.10517061300001, 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042331611999999998, 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3322 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635294+0000", 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635295+0000", 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2899999999999999e-07 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635297+0000", 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3719999999999999e-06 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635303+0000", 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4659999999999998e-06 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635306+0000", 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4800000000000001e-06 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671372+0000", 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036066415999999997 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.671503+0000", 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013060900000000001 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672027+0000", 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052349600000000003 2024-05-06T00:30:14.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677029+0000", 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050023480000000002 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677506+0000", 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047703200000000001 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677572+0000", 2024-05-06T00:30:14.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5647000000000004e-05 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677626+0000", 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4416999999999998e-05 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3323 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.105152277, 2024-05-06T00:30:14.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042505802000000002, 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3323 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635313+0000", 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2600000000000001e-07 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635316+0000", 2024-05-06T00:30:14.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.982e-06 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635320+0000", 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9319999999999998e-06 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635323+0000", 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9960000000000002e-06 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672820+0000", 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037497008999999998 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.672928+0000", 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010822700000000001 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673293+0000", 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000364526 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677035+0000", 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003742146 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677709+0000", 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067436799999999995 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677782+0000", 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2513000000000002e-05 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677819+0000", 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6876999999999999e-05 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3324 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.10513627500001, 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042718669000000001, 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3324 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635329+0000", 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4600000000000001e-07 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635331+0000", 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0379999999999998e-06 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635335+0000", 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9659999999999998e-06 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635339+0000", 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7069999999999998e-06 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673646+0000", 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038307737000000001 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673711+0000", 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4162e-05 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.673998+0000", 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028761100000000001 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677042+0000", 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030441880000000002 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677880+0000", 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083801100000000005 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677949+0000", 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8987999999999997e-05 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.678047+0000", 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8015000000000003e-05 2024-05-06T00:30:14.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4863.0:3325 3.d 3:b17d50d3:::rbd_object_map.110cf57807c6:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.105049912, 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044758777, 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4863", 2024-05-06T00:30:14.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3435535641", 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3325 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635415+0000", 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635416+0000", 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2900000000000002e-07 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635420+0000", 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3309999999999997e-06 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635430+0000", 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8390000000000002e-06 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.635439+0000", 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8910000000000001e-06 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674347+0000", 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038907820000000003 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674412+0000", 2024-05-06T00:30:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4936000000000003e-05 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.674768+0000", 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035620399999999999 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.677048+0000", 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022800860000000002 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:14.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680093+0000", 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030450299999999998 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680139+0000", 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5967999999999998e-05 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:27:18.680174+0000", 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4842999999999998e-05 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1779 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [read 12582912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 55.925799666000003, 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857773900000002, 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1779 2024-05-06T00:30:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814665+0000", 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814668+0000", 2024-05-06T00:30:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4099999999999998e-06 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814677+0000", 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4069999999999999e-06 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814700+0000", 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2469000000000001e-05 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814722+0000", 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814855+0000", 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013351500000000001 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814872+0000", 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7183000000000001e-05 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.812763+0000", 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99789039000000002 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813015+0000", 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025184099999999999 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813243+0000", 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022833400000000001 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.849 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5762.0:1780 3.d 3:b1408d66:::rbd_data.110c3ee25930.0000000000000004:head [sparse-read 12587008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 55.925699194000003, 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99922042799999999, 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5762", 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2735326789", 2024-05-06T00:30:14.850 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1780 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814766+0000", 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814767+0000", 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814783+0000", 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.594e-05 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814806+0000", 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.287e-05 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814823+0000", 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.677e-05 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814938+0000", 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115365 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:18.814959+0000", 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0786000000000001e-05 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813844+0000", 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99888496199999999 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813901+0000", 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6561000000000003e-05 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:19.813986+0000", 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5643999999999995e-05 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4364.0:27247 3.11 3:8c17c874:::rbd_object_map.110c5c796250:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.941014192000001, 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.84328285300000005, 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4364", 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 27247 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799451+0000", 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799453+0000", 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0980000000000001e-06 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799462+0000", 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8859999999999993e-06 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799481+0000", 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9239e-05 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799503+0000", 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2186000000000001e-05 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799607+0000", 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103436 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:00.799621+0000", 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:30:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.641323+0000", 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.84170262200000001 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.641474+0000", 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015097 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642589+0000", 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011146859999999999 2024-05-06T00:30:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642646+0000", 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6965000000000001e-05 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:30:01.642734+0000", 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7828e-05 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:14.860 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:14.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:30:14.863 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:14.860+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:14.962 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:14.961+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:15.034 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:15.034 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:15.034 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:30:15.034 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:15.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:30:15.093 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:15.093+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:15.164 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:15.161+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:15.217 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:15.217 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:15.217 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:30:15.217 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:30:15.217 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:15.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:30:15.273 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:15.273+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:15.367 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:15.364+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:15.367 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:30:15.400 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:30:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:30:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:30:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:30:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:15.400 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.437162351, 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:30:15.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.435977736, 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:30:15.406 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:30:15.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:15.410 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.552556690999999, 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:30:15.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:30:15.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.414 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.552519478000001, 2024-05-06T00:30:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:30:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.415 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:30:15.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:30:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:30:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:30:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:30:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.552430672, 2024-05-06T00:30:15.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.422 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:15.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:30:15.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.426 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.55237048, 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.427 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.431 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.552316177000002, 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:15.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:30:15.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.552249537000002, 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:30:15.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:30:15.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.552187265000001, 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.440 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.552128269000001, 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:30:15.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.545723547999998, 2024-05-06T00:30:15.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:15.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.545674814000002, 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:30:15.454 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:15.458 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.545630561999999, 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:15.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.545559529999998, 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.463 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:30:15.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:30:15.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.545508714999997, 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.467 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:30:15.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:15.471 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.536769567, 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:30:15.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.536746553, 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.476 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:30:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:30:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.536724303, 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:30:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.536614452999999, 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.536550403, 2024-05-06T00:30:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:15.494 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:15.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:30:15.496 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:15.492+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:15.521 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:15.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:15.538 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:15.554 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:15.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:15.577 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:15.577 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:15.596 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:15.596+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:15.597 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:15.598 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:15.601 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:15.612 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:15.612 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:15.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:30:15.697 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:15.697+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:15.783 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:15.783 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:15.798 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:15.797+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:15.799 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:30:15.899 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:15.897+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:15.905 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:15.949 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:30:16.036 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:16.036+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:16.100 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:16.100+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:16.123 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:16.123 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:16.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:30:16.228 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:16.228+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:16.297 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:16.297 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:16.302 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:16.300+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:16.311 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:30:16.402 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:16.401+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:16.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:30:16.536 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:16.536+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:16.604 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:16.604+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:16.614 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:16.614 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:16.628 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:30:16.706 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:16.705+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:16.795 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:16.795 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:16.807 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:16.805+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:16.809 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:30:16.885 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 0, 5, 3, 2] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5] 2024-05-06T00:30:16.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-06T00:30:16.885 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-05-06T00:30:16.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-05-06T00:30:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:30:16.918 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:16.916+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:16.935 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:16.935 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:16.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:30:17.008 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:17.005+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:17.086 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:17.086 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:17.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:30:17.145 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:17.145+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:17.206 INFO:teuthology.orchestra.run.smithi078.stdout:moved 0 / 82 (0%) 2024-05-06T00:30:17.206 INFO:teuthology.orchestra.run.smithi078.stdout:avg 16.4 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.64417 -> 3.64417 (expected baseline 3.62215) 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 10 -> 10 pgs (0.609756 -> 0.609756 * mean) 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.3 with 17 -> 17 pgs (1.03659 -> 1.03659 * mean) 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout:oload 1000 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 1 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0006 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0063 2024-05-06T00:30:17.207 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-06T00:30:17.210 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:17.209+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:17.225 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-05-06T00:30:17.225 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-05-06T00:30:17.268 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:17.268 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:17.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:30:17.325 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:17.325+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:17.412 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:17.409+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:17.446 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:17.447 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:17.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout:moved 0 / 82 (0%) 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout:avg 16.4 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.64417 -> 3.64417 (expected baseline 3.62215) 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 10 -> 10 pgs (0.609756 -> 0.609756 * mean) 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.3 with 17 -> 17 pgs (1.03659 -> 1.03659 * mean) 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout:oload 1000 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 0.05 2024-05-06T00:30:17.517 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-06T00:30:17.518 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0006 2024-05-06T00:30:17.518 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0063 2024-05-06T00:30:17.518 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-06T00:30:17.536 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-05-06T00:30:17.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-05-06T00:30:17.552 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:17.553+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:17.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-06T00:30:17.625 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:17.625 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:17.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:30:17.648 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:17.648+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:17.714 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:17.712+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:17.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:17.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:17.759 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:17.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:17.775 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:17.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:17.794 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:17.794 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:17.806 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:17.806 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:17.814 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:17.814 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:17.814 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:17.812+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:17.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:17.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:30:17.821 INFO:teuthology.orchestra.run.smithi078.stdout:moved 0 / 82 (0%) 2024-05-06T00:30:17.821 INFO:teuthology.orchestra.run.smithi078.stdout:avg 16.4 2024-05-06T00:30:17.821 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.64417 -> 3.64417 (expected baseline 3.62215) 2024-05-06T00:30:17.821 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 10 -> 10 pgs (0.609756 -> 0.609756 * mean) 2024-05-06T00:30:17.821 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.3 with 17 -> 17 pgs (1.03659 -> 1.03659 * mean) 2024-05-06T00:30:17.821 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:17.822 INFO:teuthology.orchestra.run.smithi078.stdout:oload 1000 2024-05-06T00:30:17.822 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 1 2024-05-06T00:30:17.822 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-06T00:30:17.822 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0006 2024-05-06T00:30:17.822 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0063 2024-05-06T00:30:17.822 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-06T00:30:17.837 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-05-06T00:30:17.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-05-06T00:30:17.915 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:17.913+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1173 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [read 1703936~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.989975199, 2024-05-06T00:30:17.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0078401459999999, 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1173 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961251+0000", 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961253+0000", 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6970000000000002e-06 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961263+0000", 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4760000000000004e-06 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961284+0000", 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1497000000000001e-05 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961301+0000", 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.696e-05 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961433+0000", 2024-05-06T00:30:17.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132285 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.961445+0000", 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1960000000000001e-05 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967390+0000", 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059449389999999998 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:17.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967413+0000", 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2943e-05 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.968956+0000", 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0015424939999999 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969091+0000", 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134895 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.956 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5628.0:1174 3.f 3:f14e6ca5:::rbd_data.110c49568cbe.0000000000000012:head [sparse-read 1708032~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.988790584, 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0070870569999999, 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5628", 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/2654899788", 2024-05-06T00:30:17.957 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1174 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962435+0000", 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962437+0000", 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6840000000000001e-06 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962445+0000", 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8930000000000005e-06 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962472+0000", 2024-05-06T00:30:17.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7287000000000001e-05 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.962492+0000", 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9933e-05 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967495+0000", 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050026580000000001 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:23.967510+0000", 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5495999999999999e-05 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969429+0000", 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.001918383 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:28:24.969522+0000", 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3722999999999997e-05 2024-05-06T00:30:17.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1442 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 24772608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.105369538999994, 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048412360000000002, 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.962 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1442 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845856+0000", 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845857+0000", 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4300000000000002e-07 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845861+0000", 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8039999999999999e-06 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845870+0000", 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7960000000000001e-06 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845881+0000", 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1077000000000001e-05 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894056+0000", 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048175257999999999 2024-05-06T00:30:17.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894186+0000", 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013004999999999999 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894269+0000", 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2632000000000004e-05 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1443 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 25821184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.105332325999996, 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048548961000000002, 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1443 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845893+0000", 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845894+0000", 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8400000000000004e-07 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845897+0000", 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.033e-06 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845909+0000", 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845922+0000", 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2923e-05 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894319+0000", 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048397432999999997 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894373+0000", 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4184999999999997e-05 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894442+0000", 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9083000000000002e-05 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.970 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1444 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 26869760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.105243520000002, 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048692881, 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1444 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845982+0000", 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845984+0000", 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3599999999999999e-06 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.845990+0000", 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846008+0000", 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7793000000000001e-05 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846022+0000", 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.428e-05 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894486+0000", 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048464144000000001 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894577+0000", 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0802000000000002e-05 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894675+0000", 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8332000000000007e-05 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.974 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1445 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 27918336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.105183327999995, 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048790965999999998, 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.975 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1445 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846042+0000", 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846044+0000", 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3370000000000001e-06 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846049+0000", 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5829999999999997e-06 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846066+0000", 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6946999999999999e-05 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846080+0000", 2024-05-06T00:30:17.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3393e-05 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894718+0000", 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048638372999999999 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894767+0000", 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8615000000000001e-05 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894833+0000", 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6717999999999994e-05 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1446 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28966912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.105129024999997, 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048953008999999999, 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1446 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846097+0000", 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846098+0000", 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.097e-06 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846103+0000", 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0830000000000003e-06 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846119+0000", 2024-05-06T00:30:17.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6405999999999998e-05 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846136+0000", 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6217000000000001e-05 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894877+0000", 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048741826000000002 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.894955+0000", 2024-05-06T00:30:17.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7676999999999995e-05 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895050+0000", 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4703000000000005e-05 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1447 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30015488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.105062384999997, 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049047273000000002, 2024-05-06T00:30:17.983 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1447 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846163+0000", 2024-05-06T00:30:17.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846165+0000", 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.13e-06 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846170+0000", 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6400000000000002e-06 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846192+0000", 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1846000000000001e-05 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846208+0000", 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5729000000000001e-05 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895094+0000", 2024-05-06T00:30:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048885980000000002 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895142+0000", 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8736000000000002e-05 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895211+0000", 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8212000000000004e-05 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.987 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1448 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31064064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.105000113000003, 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049201139999999997, 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1448 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:17.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846226+0000", 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846227+0000", 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.153e-06 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846232+0000", 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4199999999999998e-06 2024-05-06T00:30:17.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846253+0000", 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0737000000000002e-05 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846267+0000", 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4049999999999999e-05 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895258+0000", 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048990921999999999 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895343+0000", 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4698999999999997e-05 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895427+0000", 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4159000000000005e-05 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1449 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32112640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:17.991 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.104941116999996, 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049283946000000002, 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1449 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846285+0000", 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846286+0000", 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5099999999999999e-06 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846292+0000", 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.553e-06 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846309+0000", 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7493e-05 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.846323+0000", 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3487e-05 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895460+0000", 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049137551000000002 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895506+0000", 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6063000000000003e-05 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.895569+0000", 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2288999999999999e-05 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1655 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 19660800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.098536396, 2024-05-06T00:30:17.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048216883000000002, 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1655 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852689+0000", 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852690+0000", 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.108e-06 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:17.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852695+0000", 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3440000000000003e-06 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852712+0000", 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852723+0000", 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1412e-05 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:17.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900715+0000", 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047991520000000003 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900801+0000", 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6536e-05 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900906+0000", 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104935 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:17.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1656 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 20709376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.098487661999997, 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048329023999999998, 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1656 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852738+0000", 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852739+0000", 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0079999999999999e-06 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852744+0000", 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.549e-06 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852757+0000", 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3757999999999999e-05 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852768+0000", 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0572e-05 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900947+0000", 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048179166000000002 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.900998+0000", 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0912000000000003e-05 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901067+0000", 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9059000000000004e-05 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1657 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 21757952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.098443410000002, 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048533536000000002, 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1657 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852782+0000", 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852783+0000", 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.95e-07 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852788+0000", 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.011e-06 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852802+0000", 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4117000000000001e-05 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852813+0000", 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0951999999999999e-05 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901114+0000", 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048300986999999997 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901212+0000", 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7102000000000006e-05 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901316+0000", 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104372 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1658 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 22806528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.098372377999993, 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048614202000000002, 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.008 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1658 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852853+0000", 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852854+0000", 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130000000000001e-06 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852860+0000", 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7969999999999997e-06 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852876+0000", 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5424999999999999e-05 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852887+0000", 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1085e-05 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901357+0000", 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048469878000000001 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901405+0000", 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8310999999999999e-05 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901468+0000", 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2693000000000004e-05 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1659 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 23855104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.098321562999999, 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048806793000000001, 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1659 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852904+0000", 2024-05-06T00:30:18.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852905+0000", 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852911+0000", 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4929999999999998e-06 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852925+0000", 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3828999999999999e-05 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.852936+0000", 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1243000000000001e-05 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901507+0000", 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048571247999999997 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901595+0000", 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7881999999999996e-05 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.901711+0000", 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116116 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.016 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1882 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 28180480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.089582414999995, 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048368793, 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1882 2024-05-06T00:30:18.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861643+0000", 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861644+0000", 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.89e-07 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861646+0000", 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5349999999999999e-06 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861653+0000", 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8929999999999999e-06 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861659+0000", 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1789999999999996e-06 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909804+0000", 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048144345999999998 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.909897+0000", 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3146000000000006e-05 2024-05-06T00:30:18.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910012+0000", 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011520500000000001 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1883 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 29229056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.089559401000002, 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048575207000000002, 2024-05-06T00:30:18.021 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1883 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861666+0000", 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861667+0000", 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861669+0000", 2024-05-06T00:30:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-06 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861676+0000", 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6449999999999999e-06 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861682+0000", 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0920000000000003e-06 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910064+0000", 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048382149999999999 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910160+0000", 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6123000000000001e-05 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910242+0000", 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1201999999999999e-05 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1884 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 30277632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.089537151000002, 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048834057, 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1884 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861689+0000", 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6499999999999999e-07 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861692+0000", 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5220000000000002e-06 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861699+0000", 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2749999999999998e-06 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861709+0000", 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.747e-06 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910292+0000", 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048583029999999999 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910393+0000", 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010119999999999999 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910523+0000", 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012971799999999999 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1885 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 31326208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.089427301000001, 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048909541000000001, 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1885 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861798+0000", 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861800+0000", 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5740000000000001e-06 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861808+0000", 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9310000000000008e-06 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861827+0000", 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8933999999999999e-05 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861838+0000", 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226000000000001e-05 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910582+0000", 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048744096000000001 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910638+0000", 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6164e-05 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910708+0000", 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.5693.0:1886 3.1d 3:b845a099:::rbd_data.110c9161af6c.0000000000000004:28 [sparse-read 32374784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.089363250999995, 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049134651000000001, 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-05-06T00:30:18.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.5693", 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.186:0/3288471798", 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1886 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861863+0000", 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-05-06T00:30:18.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861864+0000", 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.015e-06 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861867+0000", 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7400000000000002e-06 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861880+0000", 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2319e-05 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.861887+0000", 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.109e-06 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910755+0000", 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048868502000000001 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-05-06T00:30:18.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910852+0000", 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7047999999999996e-05 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-05-06T00:29:13.910997+0000", 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014491800000000001 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-06T00:30:18.038 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:18.040 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-06T00:30:18.040 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-06T00:30:18.036+0000 7f8d481aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23996) UID: 0 2024-05-06T00:30:18.141 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:18.140+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:18.155 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:18.155 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:18.155 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:18.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:moved 0 / 82 (0%) 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:avg 16.4 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.64417 -> 3.64417 (expected baseline 3.62215) 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 10 -> 10 pgs (0.609756 -> 0.609756 * mean) 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.3 with 17 -> 17 pgs (1.03659 -> 1.03659 * mean) 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:oload 1000 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 0.05 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0006 2024-05-06T00:30:18.185 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0061 2024-05-06T00:30:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-06T00:30:18.203 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-05-06T00:30:18.203 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-05-06T00:30:18.248 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:18.248+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:18.307 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:18.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:18.308 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:18.308 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:18.308 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:18.321 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-06T00:30:18.342 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:18.341+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:1773 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 139651801587744 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 199.627774415, 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5219372610000002, 2024-05-06T00:30:18.433 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1773 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804359+0000", 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804360+0000", 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804365+0000", 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2819999999999998e-06 2024-05-06T00:30:18.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0642000000000001e-05 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804385+0000", 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7019999999999996e-06 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805254+0000", 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086896300000000005 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805260+0000", 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999998e-06 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.320794+0000", 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.515534095 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321001+0000", 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020653300000000001 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323535+0000", 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025339669999999998 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.323709+0000", 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017479799999999999 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326297+0000", 2024-05-06T00:30:18.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002587144 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4578.0:2526 3.6 3:645d64d7:::rbd_header.110cc8beeb04:head [watch reconnect cookie 140052307308096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 199.627758212, 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5222244520000001, 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4578", 2024-05-06T00:30:18.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/406428781", 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2526 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804375+0000", 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804376+0000", 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3200000000000004e-07 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804380+0000", 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9779999999999999e-06 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804389+0000", 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3979999999999999e-06 2024-05-06T00:30:18.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.804394+0000", 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7590000000000003e-06 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805274+0000", 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087936399999999999 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:26:58.805278+0000", 2024-05-06T00:30:18.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0439999999999997e-06 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321065+0000", 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5157867889999999 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.321212+0000", 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014714599999999999 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324334+0000", 2024-05-06T00:30:18.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031226790000000002 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.324488+0000", 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153893 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:00.326600+0000", 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00211167 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34116 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 864256~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.619527995, 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0739114080000001, 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34116 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812606+0000", 2024-05-06T00:30:18.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999999e-07 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812608+0000", 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3350000000000001e-06 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812616+0000", 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3250000000000002e-06 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812621+0000", 2024-05-06T00:30:18.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7440000000000004e-06 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818006+0000", 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053847210000000003 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818013+0000", 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1060000000000002e-06 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886263+0000", 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068249494 2024-05-06T00:30:18.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886400+0000", 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137358 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886517+0000", 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116965 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34117 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 870400~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.449 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61949662500001, 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0741891100000001, 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34117 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812637+0000", 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812640+0000", 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2350000000000002e-06 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812646+0000", 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4799999999999998e-06 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812651+0000", 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8199999999999996e-06 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818035+0000", 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053843229999999999 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818039+0000", 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7019999999999999e-06 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886572+0000", 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068532619 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886687+0000", 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011568100000000001 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886826+0000", 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013886599999999999 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34118 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 876544~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:18.454 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.619462836, 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074347779, 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34118 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:18.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812671+0000", 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7e-07 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812673+0000", 2024-05-06T00:30:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2390000000000001e-06 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812680+0000", 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7610000000000002e-06 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812685+0000", 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9479999999999999e-06 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818053+0000", 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053678579999999997 2024-05-06T00:30:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818056+0000", 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3950000000000001e-06 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886876+0000", 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0688193370000001 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.886949+0000", 2024-05-06T00:30:18.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3343999999999994e-05 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887019+0000", 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9527e-05 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34119 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 882688~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61928237999999, 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0744246749999999, 2024-05-06T00:30:18.459 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34119 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812851+0000", 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812852+0000", 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812855+0000", 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.636e-06 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812866+0000", 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0394000000000001e-05 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812875+0000", 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6570000000000003e-06 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818070+0000", 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051958259999999997 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818074+0000", 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3610000000000001e-06 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887061+0000", 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.068987183 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887155+0000", 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3771999999999996e-05 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887276+0000", 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121292 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34120 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 888832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61924873699999, 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.074715528, 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.464 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34120 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812885+0000", 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812888+0000", 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5409999999999999e-06 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812896+0000", 2024-05-06T00:30:18.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000007e-06 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812902+0000", 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8719999999999999e-06 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818087+0000", 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051854240000000001 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818091+0000", 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3780000000000001e-06 2024-05-06T00:30:18.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887357+0000", 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0692664670000001 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887486+0000", 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012840599999999999 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887600+0000", 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114941 2024-05-06T00:30:18.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34121 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 894976~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61922495499999, 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0750618320000001, 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34121 2024-05-06T00:30:18.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812909+0000", 2024-05-06T00:30:18.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6800000000000001e-07 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812917+0000", 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.447e-06 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812924+0000", 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.294e-06 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812929+0000", 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3750000000000002e-06 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818104+0000", 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051747779999999997 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818107+0000", 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887672+0000", 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069564993 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887841+0000", 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000168364 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.887971+0000", 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012977900000000001 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34122 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 901120~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.619198678, 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0752747250000001, 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34122 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812935+0000", 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1199999999999999e-07 2024-05-06T00:30:18.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812937+0000", 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812942+0000", 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2530000000000004e-06 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812948+0000", 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4299999999999997e-06 2024-05-06T00:30:18.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818121+0000", 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051729000000000002 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818124+0000", 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4130000000000002e-06 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888041+0000", 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.069916901 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888134+0000", 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2516999999999995e-05 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888210+0000", 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6092000000000003e-05 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.478 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34123 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 907264~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61915543699999, 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0755368139999999, 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34123 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812978+0000", 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812979+0000", 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6699999999999999e-07 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812981+0000", 2024-05-06T00:30:18.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1119999999999999e-06 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812987+0000", 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9989999999999997e-06 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.812991+0000", 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6190000000000002e-06 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818138+0000", 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051463409999999996 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818141+0000", 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888293+0000", 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0701516680000001 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888427+0000", 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013392100000000001 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888515+0000", 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8270999999999996e-05 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34124 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 913408~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61912580800001, 2024-05-06T00:30:18.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0757736250000001, 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34124 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813008+0000", 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813010+0000", 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2120000000000002e-06 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813016+0000", 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8989999999999998e-06 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813021+0000", 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6210000000000003e-06 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818154+0000", 2024-05-06T00:30:18.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005133444 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818158+0000", 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4340000000000001e-06 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888558+0000", 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0703996609999999 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888676+0000", 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011835100000000001 2024-05-06T00:30:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888781+0000", 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105624 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34126 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 919552~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61909295800001, 2024-05-06T00:30:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076063003, 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34126 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813041+0000", 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4999999999999998e-07 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813043+0000", 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1959999999999998e-06 2024-05-06T00:30:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813049+0000", 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1970000000000001e-06 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813055+0000", 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1580000000000004e-06 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818171+0000", 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005116514 2024-05-06T00:30:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818175+0000", 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5030000000000002e-06 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888834+0000", 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.070659816 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.888968+0000", 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000133108 2024-05-06T00:30:18.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889104+0000", 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013616100000000001 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34127 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 925696~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61906131999999, 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0762872210000001, 2024-05-06T00:30:18.493 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34127 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813072+0000", 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813073+0000", 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813075+0000", 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5579999999999999e-06 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813082+0000", 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7580000000000003e-06 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813087+0000", 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3469999999999998e-06 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818188+0000", 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051003200000000002 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818191+0000", 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3689999999999998e-06 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889160+0000", 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0709692259999999 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889233+0000", 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3040999999999994e-05 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889360+0000", 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012622999999999999 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34128 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 931840~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61902851799999, 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076546486, 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34128 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813105+0000", 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813106+0000", 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999999e-07 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813108+0000", 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.79e-06 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813116+0000", 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8499999999999994e-06 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813122+0000", 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3569999999999997e-06 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818204+0000", 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050827060000000002 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818208+0000", 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889391+0000", 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071183786 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889510+0000", 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118483 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889652+0000", 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000141681 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34130 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 937984~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.618996485, 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076735306, 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34130 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813137+0000", 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813138+0000", 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813140+0000", 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6740000000000001e-06 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813148+0000", 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1659999999999997e-06 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813153+0000", 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9250000000000003e-06 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818221+0000", 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005068332 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818224+0000", 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.455e-06 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889719+0000", 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0714950889999999 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889800+0000", 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1025000000000004e-05 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889872+0000", 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2076000000000006e-05 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34131 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 944128~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.618964406, 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.076921665, 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34131 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813169+0000", 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813170+0000", 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813172+0000", 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.661e-06 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813179+0000", 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0190000000000001e-06 2024-05-06T00:30:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813184+0000", 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0329999999999999e-06 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818240+0000", 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050558160000000003 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.818244+0000", 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6200000000000001e-06 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.889936+0000", 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.071691696 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890011+0000", 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5715000000000003e-05 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890091+0000", 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9698999999999997e-05 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:34132 3.12 3:4aad63bc:::rbd_data.110cf097386.0000000000000013:19 [read 950272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 133.61893309800001, 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0771274049999999, 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34132 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.813201+0000", 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828550+0000", 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0153492829999999 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828559+0000", 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2560000000000008e-06 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828577+0000", 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8043999999999999e-05 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.828595+0000", 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7601999999999999e-05 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890142+0000", 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061546998999999998 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890225+0000", 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2908e-05 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:05.890328+0000", 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010331300000000001 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4364.0:17488 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 139651801624368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 82.778787381000001, 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0820656319999999, 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4364", 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/652518431", 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 17488 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653346+0000", 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653348+0000", 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1600000000000001e-06 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653358+0000", 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8290000000000003e-06 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653380+0000", 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1934999999999999e-05 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653396+0000", 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5678000000000001e-05 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653505+0000", 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010916699999999999 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.653514+0000", 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1710000000000008e-06 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654466+0000", 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000951453 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654476+0000", 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0591000000000001e-05 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.732830+0000", 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.078353455 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.733114+0000", 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028417900000000002 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735412+0000", 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022980140000000001 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:145 3.18 3:1d65515b:::rbd_header.110cc7654932:head [call rbd.set_size in=11b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 82.777381231999996, 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.081141406, 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 145 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.523 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654752+0000", 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654754+0000", 2024-05-06T00:30:18.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.511e-06 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654761+0000", 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4749999999999996e-06 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654776+0000", 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4144999999999999e-05 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654785+0000", 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3749999999999992e-06 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654985+0000", 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000199672 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654990+0000", 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.733e-06 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734015+0000", 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.07902513 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734150+0000", 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000134213 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.734439+0000", 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028922799999999999 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735477+0000", 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010380789999999999 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735710+0000", 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023328699999999999 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735782+0000", 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2115999999999999e-05 2024-05-06T00:30:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735894+0000", 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011144199999999999 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5542.0:146 3.18 3:1d65515b:::rbd_header.110cc7654932:head [watch reconnect cookie 140709235982672 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 82.777328273999998, 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0827949809999999, 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5542", 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/47451673", 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 146 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654805+0000", 2024-05-06T00:30:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654818+0000", 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2432999999999999e-05 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654822+0000", 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654835+0000", 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-06T00:30:18.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.654843+0000", 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9060000000000008e-06 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655573+0000", 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072929099999999997 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:55.655587+0000", 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3937000000000001e-05 2024-05-06T00:30:18.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735011+0000", 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794246670000001 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735195+0000", 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184155 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.735956+0000", 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076100799999999998 2024-05-06T00:30:18.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.736052+0000", 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5149000000000007e-05 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:56.737600+0000", 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001548743 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:18.534 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:18.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-06T00:30:18.536 INFO:teuthology.orchestra.run.smithi078.stdout:moved 0 / 82 (0%) 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:avg 16.4 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.64417 -> 3.64417 (expected baseline 3.62215) 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 10 -> 10 pgs (0.609756 -> 0.609756 * mean) 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.3 with 17 -> 17 pgs (1.03659 -> 1.03659 * mean) 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:oload 1000 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 0.05 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0006 2024-05-06T00:30:18.537 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0061 2024-05-06T00:30:18.538 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-06T00:30:18.538 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:18.533+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:18.637 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:18.636+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:18.658 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:18.658 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:18.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-06T00:30:18.739 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:18.737+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:18.839 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:18.836+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:18.840 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:18.840 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:18.840 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:18.840 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:18.840 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:18.854 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-06T00:30:18.942 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-06T00:30:18.940+0000 7f1eb02dc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23994) UID: 0 2024-05-06T00:30:18.951 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-06T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21574 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 442368~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.006340338, 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090838402999999998, 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21574 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:19.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027133+0000", 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027135+0000", 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0779999999999998e-06 2024-05-06T00:30:19.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027141+0000", 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027146+0000", 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6929999999999998e-06 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117830+0000", 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090684497000000003 2024-05-06T00:30:19.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117888+0000", 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7340000000000003e-05 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117971+0000", 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3363999999999997e-05 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.037 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21576 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 448512~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00631954900001, 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090988716999999997, 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21576 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027153+0000", 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027154+0000", 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027156+0000", 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1679999999999998e-06 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027162+0000", 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2530000000000001e-06 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027167+0000", 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9860000000000002e-06 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.117998+0000", 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090830533000000005 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118057+0000", 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9382999999999997e-05 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118142+0000", 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4856000000000002e-05 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21578 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 454656~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00629948700001, 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091782412999999993, 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21578 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.042 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027173+0000", 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027174+0000", 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-06T00:30:19.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027176+0000", 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0150000000000002e-06 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027182+0000", 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.896e-06 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027187+0000", 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8980000000000004e-06 2024-05-06T00:30:19.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118169+0000", 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090982705999999997 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118851+0000", 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068203999999999997 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118956+0000", 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010445100000000001 2024-05-06T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21580 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 460800~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00628021099999, 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093391094999999993, 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21580 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027193+0000", 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027195+0000", 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9939999999999999e-06 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027201+0000", 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3199999999999996e-06 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027207+0000", 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.118991+0000", 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091783951000000003 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.119576+0000", 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058497900000000001 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120584+0000", 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010082330000000001 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21582 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 466944~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00626009300001, 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095303559999999996, 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21582 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027213+0000", 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.46e-07 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027215+0000", 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.153e-06 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027222+0000", 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1430000000000003e-06 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027237+0000", 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5495e-05 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.120619+0000", 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.093382218000000003 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.121025+0000", 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040603500000000001 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122516+0000", 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014910699999999999 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21584 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 473088~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00622868400001, 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095167369000000002, 2024-05-06T00:30:19.054 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21584 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027244+0000", 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027245+0000", 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6400000000000003e-07 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027252+0000", 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7729999999999999e-06 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027259+0000", 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0169999999999999e-06 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027270+0000", 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122149+0000", 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094879079000000005 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122252+0000", 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102845 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122412+0000", 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015959500000000001 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21586 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 479232~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:19.058 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00619577200001, 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095419918000000006, 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21586 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027277+0000", 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027278+0000", 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027280+0000", 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2970000000000002e-06 2024-05-06T00:30:19.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027286+0000", 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1090000000000003e-06 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027291+0000", 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122439+0000", 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095147624 2024-05-06T00:30:19.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122541+0000", 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101804 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122697+0000", 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015651099999999999 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.062 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21588 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 485376~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00617518499999, 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095822849000000002, 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21588 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027298+0000", 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5499999999999998e-07 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027300+0000", 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.029e-06 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027306+0000", 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027311+0000", 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1510000000000003e-06 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122729+0000", 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095418119999999995 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.122791+0000", 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2100999999999994e-05 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123121+0000", 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032907599999999998 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.066 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21590 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 491520~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00615580499999, 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096094375999999995, 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21590 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.067 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027317+0000", 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.27e-07 2024-05-06T00:30:19.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027335+0000", 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7147999999999999e-05 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027347+0000", 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2396e-05 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027353+0000", 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8560000000000003e-06 2024-05-06T00:30:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123151+0000", 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.095798279 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123252+0000", 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010069300000000001 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123411+0000", 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015957699999999999 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21592 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 497664~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.006111124, 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096607112999999994, 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21592 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.071 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027362+0000", 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9500000000000003e-07 2024-05-06T00:30:19.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027364+0000", 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1519999999999999e-06 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027394+0000", 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9145999999999998e-05 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027407+0000", 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3560999999999999e-05 2024-05-06T00:30:19.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123441+0000", 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096033427000000005 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123540+0000", 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9673999999999995e-05 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.123969+0000", 2024-05-06T00:30:19.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000428658 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21594 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 503808~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.00604603299999, 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096852378000000003, 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21594 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027427+0000", 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-07 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027430+0000", 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.898e-06 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027438+0000", 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1969999999999999e-06 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027443+0000", 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3170000000000001e-06 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124005+0000", 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096562435000000002 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124114+0000", 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109195 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124279+0000", 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016476699999999999 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21596 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 509952~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.006014459, 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097120183999999998, 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.079 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21596 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027458+0000", 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027459+0000", 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027461+0000", 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3590000000000002e-06 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027468+0000", 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5660000000000003e-06 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027473+0000", 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6210000000000001e-06 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124312+0000", 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096838854000000002 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124419+0000", 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106541 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124579+0000", 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015980899999999999 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21598 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 516096~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.005992198, 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097332479999999999, 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21598 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:19.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027481+0000", 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7100000000000002e-07 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027483+0000", 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1859999999999999e-06 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027490+0000", 2024-05-06T00:30:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5729999999999996e-06 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027495+0000", 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2100000000000001e-06 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124610+0000", 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097114986 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124717+0000", 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010669400000000001 2024-05-06T00:30:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124813+0000", 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6360000000000006e-05 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:21600 3.10 3:0b283787:::rbd_data.110cf097386.0000000000000009:19 [read 522240~1024] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.005971082, 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097469921000000001, 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 21600 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:19.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027502+0000", 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027504+0000", 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.092e-06 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027510+0000", 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:30:19.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.027517+0000", 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7900000000000002e-06 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124832+0000", 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.097314911000000004 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124889+0000", 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7086000000000002e-05 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.124972+0000", 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2433000000000002e-05 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30544 3.10 3:0b763787:::rbd_data.110cfb96f133.0000000000000067:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 134.34825445600001, 2024-05-06T00:30:19.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091033188000000001, 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30544 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685218+0000", 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685243+0000", 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4952000000000001e-05 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685246+0000", 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1940000000000001e-06 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685250+0000", 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2390000000000004e-06 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685253+0000", 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.393e-06 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.764947+0000", 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079693718999999996 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.765076+0000", 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012949500000000001 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776252+0000", 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011175196 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:30553 3.10 3:0ee710aa:::rbd_data.110cfb96f133.000000000000006b:head [stat,delete] snapc 1a=[1a] RETRY=7 ondisk+retry+write+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.095 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 134.34784421399999, 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.090753932999999995, 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 30553 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685629+0000", 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685728+0000", 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9599000000000006e-05 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685732+0000", 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5559999999999999e-06 2024-05-06T00:30:19.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685740+0000", 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0460000000000006e-06 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.685749+0000", 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766273+0000", 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.080524092000000005 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.766411+0000", 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000137742 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.776383+0000", 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099720209999999993 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.099 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1012 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.409199619999995, 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012792151, 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1012 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624273+0000", 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624275+0000", 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.08e-06 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624283+0000", 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5580000000000003e-06 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624305+0000", 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2132000000000001e-05 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624320+0000", 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5474000000000001e-05 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626412+0000", 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020918830000000001 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626432+0000", 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9904000000000001e-05 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636846+0000", 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010413496 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.636960+0000", 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0001144 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637065+0000", 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010522399999999999 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1013 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.409128335999995, 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0129871539999999, 2024-05-06T00:30:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1013 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624345+0000", 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624346+0000", 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8599999999999996e-07 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624351+0000", 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1739999999999999e-06 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624360+0000", 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5519999999999993e-06 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624367+0000", 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626546+0000", 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002179187 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626569+0000", 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637114+0000", 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.010544941 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637180+0000", 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5226000000000002e-05 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637332+0000", 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015216900000000001 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1014 3.8 3:12862e01:::rbd_data.110c43386310.0000000000000000:head [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.409096703000003, 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013904414, 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1014 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624376+0000", 2024-05-06T00:30:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624377+0000", 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8599999999999998e-07 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624379+0000", 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8059999999999999e-06 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624386+0000", 2024-05-06T00:30:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.624393+0000", 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1859999999999997e-06 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626632+0000", 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022394530000000002 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.626639+0000", 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4829999999999996e-06 2024-05-06T00:30:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637905+0000", 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0112660069999999 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638068+0000", 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163833 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.638281+0000", 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212196 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5503.0:1018 3.0 3:01b3819b:::rbd_data.110c43386310.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2134016~61440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.404878156999999, 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.013266668, 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5503", 2024-05-06T00:30:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/310483215", 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1018 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628595+0000", 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628596+0000", 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.353e-06 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628634+0000", 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7540000000000003e-05 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628646+0000", 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359e-05 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628656+0000", 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0253e-05 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628738+0000", 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1756999999999997e-05 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:51.628752+0000", 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3806e-05 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637359+0000", 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0086068420000001 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637454+0000", 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5242999999999996e-05 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.637607+0000", 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152669 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.640560+0000", 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002953252 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641751+0000", 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011916540000000001 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641814+0000", 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2199999999999994e-05 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:52.641861+0000", 2024-05-06T00:30:19.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.774e-05 2024-05-06T00:30:19.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.120 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:19.122 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-06T00:30:19.122 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:19.120+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:19.223 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:19.220+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:19.284 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:19.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:19.284 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-06T00:30:19.284 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:19.298 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-06T00:30:19.340 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:19.340+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:19.425 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-05-06T00:30:19.425+0000 7fe804a3f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24052) UID: 0 2024-05-06T00:30:19.461 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:19.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-06T00:30:19.461 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-06T00:30:19.461 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-06T00:30:19.461 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:19.475 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-06T00:30:19.525 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:19.524+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:19.597 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:19.597 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-06T00:30:19.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-06T00:30:19.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:924 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 171.59181936900001, 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058778389, 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 924 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006711+0000", 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006712+0000", 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.144e-06 2024-05-06T00:30:19.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006718+0000", 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.75e-06 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006736+0000", 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8016000000000002e-05 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006749+0000", 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3274e-05 2024-05-06T00:30:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065123+0000", 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058373345 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065180+0000", 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7195999999999999e-05 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065489+0000", 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030966400000000002 2024-05-06T00:30:19.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:925 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 171.59159079700001, 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059011363999999997, 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 925 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006940+0000", 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006941+0000", 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006950+0000", 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4239999999999993e-06 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006974+0000", 2024-05-06T00:30:19.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4579999999999998e-05 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.006992+0000", 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7473999999999999e-05 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065529+0000", 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058536745000000001 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065623+0000", 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4846999999999995e-05 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.065951+0000", 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032761800000000001 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:926 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4096000~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:19.606 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 171.591513513, 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059482189999999997, 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 926 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007017+0000", 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007018+0000", 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.44e-06 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007041+0000", 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2702999999999999e-05 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007062+0000", 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0503e-05 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007077+0000", 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5094e-05 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066003+0000", 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058926553 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066107+0000", 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103737 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066499+0000", 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039216000000000001 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.610 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4998.0:927 3.1e 3:7c87914d:::rbd_data.110c5ae73073.0000000000000003:10 [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 171.59131753200001, 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059542261999999999, 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4998", 2024-05-06T00:30:19.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/2792461625", 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 927 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007213+0000", 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007215+0000", 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-06 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007222+0000", 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5730000000000002e-06 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007245+0000", 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2773999999999999e-05 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.007262+0000", 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.685e-05 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066543+0000", 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059281291999999999 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066647+0000", 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103711 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:27:28.066755+0000", 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108402 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4036 3.d 3:b59f5504:::rbd_data.110cfb96f133.0000000000000055:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 138.68420815600001, 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079689258999999998, 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4036 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914322+0000", 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914323+0000", 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914325+0000", 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0820000000000001e-06 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914331+0000", 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914336+0000", 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7359999999999998e-06 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915562+0000", 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001225805 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915673+0000", 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011063799999999999 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994012+0000", 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078338838999999993 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:4043 3.1f 3:fc56abcb:::rbd_data.110cfb96f133.0000000000000058:head [stat,delete] snapc 1a=[1a] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-05-06T00:30:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 138.68414727699999, 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.079634943999999999, 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4043 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914383+0000", 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914384+0000", 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8000000000000001e-07 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914386+0000", 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.148e-06 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914393+0000", 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7880000000000001e-06 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.914398+0000", 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2129999999999999e-06 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915909+0000", 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015113839999999999 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.915988+0000", 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8961000000000005e-05 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:00.994018+0000", 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078030070000000007 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23608 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 825344~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.566590827, 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059540322, 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23608 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031940+0000", 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031941+0000", 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.217e-06 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031947+0000", 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0260000000000004e-06 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031952+0000", 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6820000000000002e-06 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031956+0000", 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5480000000000003e-06 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089823+0000", 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.057866888999999998 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.089920+0000", 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7408e-05 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091480+0000", 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015595520000000001 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23610 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 831488~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.56654609899999, 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059819864, 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23610 2024-05-06T00:30:19.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031984+0000", 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031986+0000", 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.223e-06 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031988+0000", 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1629999999999999e-06 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031993+0000", 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9490000000000004e-06 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.031997+0000", 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.746e-06 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091514+0000", 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059516336000000003 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091632+0000", 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118189 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091804+0000", 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172258 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23612 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 837632~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.56650769800001, 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060572269999999998, 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23612 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032023+0000", 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032024+0000", 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2020000000000001e-06 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032026+0000", 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3369999999999998e-06 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032031+0000", 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9459999999999997e-06 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032036+0000", 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.498e-06 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091830+0000", 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059794725 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.091931+0000", 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100294 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092595+0000", 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00066426799999999998 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23614 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 843776~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.636 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.566470436, 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060829657000000002, 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23614 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:19.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032060+0000", 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032061+0000", 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.336e-06 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032064+0000", 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.345e-06 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032068+0000", 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7480000000000001e-06 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032073+0000", 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3390000000000003e-06 2024-05-06T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092627+0000", 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060554519000000001 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092734+0000", 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106612 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.092890+0000", 2024-05-06T00:30:19.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575800000000001 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23616 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 849920~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.56643352699999, 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061282310999999999, 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.641 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23616 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032097+0000", 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032098+0000", 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.412e-06 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032100+0000", 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1090000000000001e-06 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032105+0000", 2024-05-06T00:30:19.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7099999999999998e-06 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032110+0000", 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093120+0000", 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061010196000000003 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093220+0000", 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9463999999999995e-05 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093379+0000", 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000159591 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.645 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23618 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 856064~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.56639560900001, 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061536879000000003, 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23618 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032135+0000", 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032136+0000", 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1620000000000001e-06 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032138+0000", 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3e-06 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032143+0000", 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8439999999999997e-06 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032147+0000", 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2570000000000001e-06 2024-05-06T00:30:19.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093410+0000", 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061262144999999997 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093505+0000", 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5533999999999997e-05 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093672+0000", 2024-05-06T00:30:19.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166637 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23620 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 862208~1024] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.56635849400001, 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061805882999999999, 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23620 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032172+0000", 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032173+0000", 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4079999999999999e-06 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032180+0000", 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1840000000000004e-06 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032184+0000", 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8960000000000002e-06 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.032189+0000", 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093709+0000", 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.061519809000000002 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093816+0000", 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107092 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.093978+0000", 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016192600000000001 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23626 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 880640~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.560578045, 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058563883999999997, 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23626 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037952+0000", 2024-05-06T00:30:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037953+0000", 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4900000000000001e-07 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037954+0000", 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.702e-06 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037961+0000", 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1199999999999999e-06 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037965+0000", 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8119999999999998e-06 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.094588+0000", 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056622748000000001 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096326+0000", 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001738114 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096516+0000", 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190039 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23628 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 886784~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.56053996099999, 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058862182999999998, 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23628 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037991+0000", 2024-05-06T00:30:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037993+0000", 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9069999999999999e-06 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.037999+0000", 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9170000000000003e-06 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038003+0000", 2024-05-06T00:30:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7319999999999998e-06 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096565+0000", 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058561167999999997 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096673+0000", 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108152 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096853+0000", 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017993800000000001 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23630 3.1f 3:fc074bce:::rbd_data.110cf097386.0000000000000002:19 [read 892928~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.56050228699999, 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.059119786000000001, 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23630 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038028+0000", 2024-05-06T00:30:19.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038029+0000", 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8700000000000001e-07 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038031+0000", 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.04e-06 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038036+0000", 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8839999999999999e-06 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.038041+0000", 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9740000000000001e-06 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096885+0000", 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058843233000000002 2024-05-06T00:30:19.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.096985+0000", 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9926000000000005e-05 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.097148+0000", 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163342 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23719 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.55106677699999, 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473018000000002, 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23719 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047464+0000", 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-05-06T00:30:19.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047466+0000", 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9439999999999999e-06 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047474+0000", 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047488+0000", 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3723000000000001e-05 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105924+0000", 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058436258999999997 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105937+0000", 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.671 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23720 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.55103435000001, 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058476526000000001, 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23720 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047496+0000", 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047497+0000", 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047499+0000", 2024-05-06T00:30:19.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2359999999999999e-06 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047507+0000", 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0209999999999999e-06 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047514+0000", 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6669999999999999e-06 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105962+0000", 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058448884999999999 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.105973+0000", 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0198e-05 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.676 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23722 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.55101046799999, 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058500451000000002, 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.677 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23722 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047520+0000", 2024-05-06T00:30:19.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047521+0000", 2024-05-06T00:30:19.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1500000000000004e-07 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047523+0000", 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8980000000000001e-06 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047529+0000", 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0870000000000003e-06 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047533+0000", 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.843e-06 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106007+0000", 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058473962999999997 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106020+0000", 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3045e-05 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.5155.0:23724 3.1f 3.73d2e03f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 135.55099053800001, 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058509541999999998, 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.5155", 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.186:0/786346670", 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 23724 2024-05-06T00:30:19.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-06T00:30:19.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047540+0000", 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9800000000000004e-07 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047543+0000", 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2230000000000001e-06 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047549+0000", 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.047572+0000", 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3331000000000001e-05 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106040+0000", 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.058467681000000001 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-06T00:28:04.106049+0000", 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6840000000000008e-06 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.685 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-06T00:30:19.686 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-06T00:30:19.686 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:19.687 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-06T00:30:19.688 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:19.685+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:19.788 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:19.788+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:19.819 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-06T00:30:19.819 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:19.819 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:30:19.820 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:19.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-06T00:30:19.889 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:19.889+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:19.940 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-06T00:30:19.957 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:19.958 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:19.958 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-05-06T00:30:19.958 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-05-06T00:30:19.958 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:19.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-06T00:30:19.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-06T00:30:19.980 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-06T00:30:19.989 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-06T00:30:19.988+0000 7faa8f0bc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23998) UID: 0 2024-05-06T00:30:19.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-06T00:30:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-06T00:30:20.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-06T00:30:20.015 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-06T00:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-06T00:30:20.033 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-06T00:30:20.033 INFO:teuthology.orchestra.run.smithi078.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-06T00:30:20.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-06T00:30:20.051 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-06T00:30:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-06T00:30:20.054 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-06T00:30:20.090 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-05-06T00:30:20.089+0000 7f64d8a49640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24054) UID: 0 2024-05-06T00:30:20.095 INFO:teuthology.orchestra.run.smithi186.stdout:All operations completed A-OK! 2024-05-06T00:30:20.134 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:20.134 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:20.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-06T00:30:20.173 DEBUG:teuthology.parallel:result is None 2024-05-06T00:30:20.173 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-05-06T00:30:20.183 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-05-06T00:30:20.222 INFO:tasks.thrashosds:joining thrashosds 2024-05-06T00:30:20.347 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:20.347 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:20.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-06T00:30:20.484 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:20.484 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:20.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-06T00:30:20.615 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-05-06T00:30:20.615 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-06T00:30:21.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-06T00:30:22.976 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-06T00:30:22.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-06T00:30:23.543 INFO:tasks.thrashosds.thrasher:reviving osd 2024-05-06T00:30:23.543 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-05-06T00:30:23.544 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-06T00:30:23.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-06T00:30:23.547 INFO:tasks.ceph.osd.4:Started 2024-05-06T00:30:23.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:no valid command found; 10 closest matches: 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:0 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:1 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:2 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:abort 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:assert 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:config diff 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:config diff get 2024-05-06T00:30:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:config get 2024-05-06T00:30:23.670 INFO:teuthology.orchestra.run.smithi146.stderr:config help [] 2024-05-06T00:30:23.670 INFO:teuthology.orchestra.run.smithi146.stderr:config set ... 2024-05-06T00:30:23.670 INFO:teuthology.orchestra.run.smithi146.stderr:admin_socket: invalid command 2024-05-06T00:30:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-06T00:30:23.672 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-05-06T00:30:23.982 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-06T00:30:24.155 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:30:24.153+0000 7f44281d17c0 -1 Falling back to public interface 2024-05-06T00:30:28.005 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:30:28.005+0000 7f44281d17c0 -1 osd.4 174 log_to_monitors true 2024-05-06T00:30:28.565 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-05-06T00:30:28.565+0000 7f441bfae640 -1 osd.4 174 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T00:30:28.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-06T00:30:28.804 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-06T00:30:28.805 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-05-06T00:30:28.805 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-05-06T00:30:28.805 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-06T00:30:28.819 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-05-06T00:30:28.962 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a: {} 2024-05-06T00:30:28.963 INFO:teuthology.orchestra.run.smithi078.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-06T00:30:28.975 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b: {} 2024-05-06T00:30:28.975 INFO:teuthology.orchestra.run.smithi078.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-06T00:30:28.985 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c: {} 2024-05-06T00:30:28.985 INFO:teuthology.orchestra.run.smithi078.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-06T00:30:29.002 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-05-06T00:30:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.0 to 1 (10000) 2024-05-06T00:30:29.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-05-06T00:30:30.528 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-05-06T00:30:30.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-05-06T00:30:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.1 to 1 (10000) 2024-05-06T00:30:31.582 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-05-06T00:30:32.563 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-05-06T00:30:32.586 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-05-06T00:30:33.570 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.2 to 1 (10000) 2024-05-06T00:30:33.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-05-06T00:30:34.575 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-05-06T00:30:34.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-05-06T00:30:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.3 to 1 (10000) 2024-05-06T00:30:35.607 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-05-06T00:30:36.592 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-05-06T00:30:36.618 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-05-06T00:30:37.638 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.5 to 1 (10000) 2024-05-06T00:30:37.662 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-05-06T00:30:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-05-06T00:30:38.667 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-05-06T00:30:39.642 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.4 to 1 (10000) 2024-05-06T00:30:39.670 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-05-06T00:30:40.660 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-05-06T00:30:40.686 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-05-06T00:30:40.687 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-05-06T00:30:40.687 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-05-06T00:30:40.687 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-05-06T00:30:40.687 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-06T00:30:40.687 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T00:30:40.991 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:40.991 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":210,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:30:40.653120+0000","last_up_change":"2024-05-06T00:30:29.524624+0000","last_in_change":"2024-05-06T00:28:54.626879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T00:26:28.436494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-06T00:26:33.941780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-05-06T00:26:42.622595+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6817","nonce":1427564096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6819","nonce":1427564096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6823","nonce":1427564096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6821","nonce":1427564096}]},"public_addr":"172.21.15.78:6817/1427564096","cluster_addr":"172.21.15.78:6819/1427564096","heartbeat_back_addr":"172.21.15.78:6823/1427564096","heartbeat_front_addr":"172.21.15.78:6821/1427564096","state":["exists","up"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6801","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6803","nonce":2663808859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6805","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6807","nonce":2663808859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6815","nonce":2663808859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6811","nonce":2663808859}]},"public_addr":"172.21.15.78:6803/2663808859","cluster_addr":"172.21.15.78:6807/2663808859","heartbeat_back_addr":"172.21.15.78:6815/2663808859","heartbeat_front_addr":"172.21.15.78:6811/2663808859","state":["exists","up"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6802","nonce":385938386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6806","nonce":385938386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6814","nonce":385938386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6810","nonce":385938386}]},"public_addr":"172.21.15.78:6802/385938386","cluster_addr":"172.21.15.78:6806/385938386","heartbeat_back_addr":"172.21.15.78:6814/385938386","heartbeat_front_addr":"172.21.15.78:6810/385938386","state":["exists","up"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6812","nonce":4178038317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6816","nonce":4178038317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6822","nonce":4178038317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6817","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6819","nonce":4178038317}]},"public_addr":"172.21.15.146:6812/4178038317","cluster_addr":"172.21.15.146:6816/4178038317","heartbeat_back_addr":"172.21.15.146:6822/4178038317","heartbeat_front_addr":"172.21.15.146:6819/4178038317","state":["exists","up"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":174,"up_from":198,"up_thru":198,"down_at":175,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6809","nonce":3162350242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6813","nonce":3162350242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6823","nonce":3162350242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6818","nonce":3162350242}]},"public_addr":"172.21.15.146:6809/3162350242","cluster_addr":"172.21.15.146:6813/3162350242","heartbeat_back_addr":"172.21.15.146:6823/3162350242","heartbeat_front_addr":"172.21.15.146:6818/3162350242","state":["exists","up"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6801","nonce":2019146845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6803","nonce":2019146845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6807","nonce":2019146845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6805","nonce":2019146845}]},"public_addr":"172.21.15.146:6801/2019146845","cluster_addr":"172.21.15.146:6803/2019146845","heartbeat_back_addr":"172.21.15.146:6807/2019146845","heartbeat_front_addr":"172.21.15.146:6805/2019146845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.276988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.054473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.154558+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.284746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.259847+0000","dead_epoch":175},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.180375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1","mappings":[{"from":2,"to":3}]},{"pgid":"3.6","mappings":[{"from":1,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:30:41.009 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-06T00:30:41.009 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T00:30:41.306 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:41.306 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":210,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:30:40.653120+0000","last_up_change":"2024-05-06T00:30:29.524624+0000","last_in_change":"2024-05-06T00:28:54.626879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T00:26:28.436494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-06T00:26:33.941780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-05-06T00:26:42.622595+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6817","nonce":1427564096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6819","nonce":1427564096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6823","nonce":1427564096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6821","nonce":1427564096}]},"public_addr":"172.21.15.78:6817/1427564096","cluster_addr":"172.21.15.78:6819/1427564096","heartbeat_back_addr":"172.21.15.78:6823/1427564096","heartbeat_front_addr":"172.21.15.78:6821/1427564096","state":["exists","up"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6801","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6803","nonce":2663808859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6805","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6807","nonce":2663808859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6815","nonce":2663808859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6811","nonce":2663808859}]},"public_addr":"172.21.15.78:6803/2663808859","cluster_addr":"172.21.15.78:6807/2663808859","heartbeat_back_addr":"172.21.15.78:6815/2663808859","heartbeat_front_addr":"172.21.15.78:6811/2663808859","state":["exists","up"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6802","nonce":385938386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6806","nonce":385938386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6814","nonce":385938386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6810","nonce":385938386}]},"public_addr":"172.21.15.78:6802/385938386","cluster_addr":"172.21.15.78:6806/385938386","heartbeat_back_addr":"172.21.15.78:6814/385938386","heartbeat_front_addr":"172.21.15.78:6810/385938386","state":["exists","up"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6812","nonce":4178038317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6816","nonce":4178038317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6822","nonce":4178038317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6817","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6819","nonce":4178038317}]},"public_addr":"172.21.15.146:6812/4178038317","cluster_addr":"172.21.15.146:6816/4178038317","heartbeat_back_addr":"172.21.15.146:6822/4178038317","heartbeat_front_addr":"172.21.15.146:6819/4178038317","state":["exists","up"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":174,"up_from":198,"up_thru":198,"down_at":175,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6809","nonce":3162350242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6813","nonce":3162350242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6823","nonce":3162350242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6818","nonce":3162350242}]},"public_addr":"172.21.15.146:6809/3162350242","cluster_addr":"172.21.15.146:6813/3162350242","heartbeat_back_addr":"172.21.15.146:6823/3162350242","heartbeat_front_addr":"172.21.15.146:6818/3162350242","state":["exists","up"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6801","nonce":2019146845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6803","nonce":2019146845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6807","nonce":2019146845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6805","nonce":2019146845}]},"public_addr":"172.21.15.146:6801/2019146845","cluster_addr":"172.21.15.146:6803/2019146845","heartbeat_back_addr":"172.21.15.146:6807/2019146845","heartbeat_front_addr":"172.21.15.146:6805/2019146845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.276988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.054473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.154558+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.284746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.259847+0000","dead_epoch":175},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.180375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1","mappings":[{"from":2,"to":3}]},{"pgid":"3.6","mappings":[{"from":1,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:30:41.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-06T00:30:41.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-06T00:30:41.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-06T00:30:41.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-06T00:30:41.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-06T00:30:41.326 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-06T00:30:41.487 INFO:teuthology.orchestra.run.smithi078.stdout:47244640310 2024-05-06T00:30:41.488 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-06T00:30:41.492 INFO:teuthology.orchestra.run.smithi078.stdout:47244640312 2024-05-06T00:30:41.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-06T00:30:41.547 INFO:teuthology.orchestra.run.smithi078.stdout:47244640311 2024-05-06T00:30:41.547 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-06T00:30:41.554 INFO:teuthology.orchestra.run.smithi078.stdout:850403524612 2024-05-06T00:30:41.554 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-06T00:30:41.566 INFO:teuthology.orchestra.run.smithi078.stdout:47244640310 2024-05-06T00:30:41.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-06T00:30:41.770 INFO:teuthology.orchestra.run.smithi078.stdout:47244640312 2024-05-06T00:30:41.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-06T00:30:41.841 INFO:teuthology.orchestra.run.smithi078.stdout:47244640311 2024-05-06T00:30:41.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640312 got 47244640311 for osd.5 2024-05-06T00:30:41.875 INFO:teuthology.orchestra.run.smithi078.stdout:47244640310 2024-05-06T00:30:41.892 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640311 got 47244640310 for osd.2 2024-05-06T00:30:41.926 INFO:teuthology.orchestra.run.smithi078.stdout:47244640309 2024-05-06T00:30:41.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640310 got 47244640309 for osd.3 2024-05-06T00:30:42.003 INFO:teuthology.orchestra.run.smithi078.stdout:47244640309 2024-05-06T00:30:42.024 INFO:teuthology.orchestra.run.smithi078.stdout:850403524611 2024-05-06T00:30:42.026 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640310 got 47244640309 for osd.0 2024-05-06T00:30:42.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 850403524612 got 850403524611 for osd.4 2024-05-06T00:30:42.137 INFO:teuthology.orchestra.run.smithi078.stdout:47244640311 2024-05-06T00:30:42.155 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640312 got 47244640311 for osd.1 2024-05-06T00:30:42.860 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-06T00:30:42.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-06T00:30:42.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-06T00:30:43.027 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-06T00:30:43.041 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-06T00:30:43.156 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-06T00:30:43.171 INFO:teuthology.orchestra.run.smithi078.stdout:47244640312 2024-05-06T00:30:43.191 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640312 got 47244640312 for osd.5 2024-05-06T00:30:43.194 DEBUG:teuthology.parallel:result is None 2024-05-06T00:30:43.257 INFO:teuthology.orchestra.run.smithi078.stdout:47244640311 2024-05-06T00:30:43.282 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640311 got 47244640311 for osd.2 2024-05-06T00:30:43.282 DEBUG:teuthology.parallel:result is None 2024-05-06T00:30:43.320 INFO:teuthology.orchestra.run.smithi078.stdout:47244640310 2024-05-06T00:30:43.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640310 got 47244640310 for osd.0 2024-05-06T00:30:43.338 DEBUG:teuthology.parallel:result is None 2024-05-06T00:30:43.371 INFO:teuthology.orchestra.run.smithi078.stdout:47244640310 2024-05-06T00:30:43.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640310 got 47244640310 for osd.3 2024-05-06T00:30:43.389 DEBUG:teuthology.parallel:result is None 2024-05-06T00:30:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:850403524612 2024-05-06T00:30:43.471 INFO:teuthology.orchestra.run.smithi078.stdout:47244640312 2024-05-06T00:30:43.474 INFO:tasks.ceph.ceph_manager.ceph:need seq 850403524612 got 850403524612 for osd.4 2024-05-06T00:30:43.474 DEBUG:teuthology.parallel:result is None 2024-05-06T00:30:43.488 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640312 got 47244640312 for osd.1 2024-05-06T00:30:43.489 DEBUG:teuthology.parallel:result is None 2024-05-06T00:30:43.489 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-05-06T00:30:43.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:30:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:43.816 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:30:43.828 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-05-06T00:30:42.460500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":625,"num_object_clones":0,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":619,"num_read":263813,"num_read_kb":883109,"num_write":20149,"num_write_kb":194254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5740,"num_bytes_recovered":73786338,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19817,"ondisk_log_size":19817,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":283216,"kb_used_data":15976,"kb_used_omap":52,"kb_used_meta":267083,"kb_avail":565947824,"statfs":{"total":579820584960,"available":579530571776,"internally_reserved":0,"allocated":16359424,"data_stored":7574696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53637,"internal_metadata":273493627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.824385"},"pg_stats":[{"pgid":"3.1f","version":"194'672","reported_seq":13023,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725510+0000","last_change":"2024-05-06T00:30:19.956846+0000","last_active":"2024-05-06T00:30:40.725510+0000","last_peered":"2024-05-06T00:30:40.725510+0000","last_clean":"2024-05-06T00:30:40.725510+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:30:40.725510+0000","last_undegraded":"2024-05-06T00:30:40.725510+0000","last_fullsized":"2024-05-06T00:30:40.725510+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:12:04.053936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028964080000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":12460,"num_read_kb":40065,"num_write":528,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"191'367","reported_seq":9956,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725314+0000","last_change":"2024-05-06T00:30:19.973477+0000","last_active":"2024-05-06T00:30:40.725314+0000","last_peered":"2024-05-06T00:30:40.725314+0000","last_clean":"2024-05-06T00:30:40.725314+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:30:40.725314+0000","last_undegraded":"2024-05-06T00:30:40.725314+0000","last_fullsized":"2024-05-06T00:30:40.725314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:17.412771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012177069999999999,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":8762,"num_read_kb":50019,"num_write":296,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"199'456","reported_seq":4006,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:30.637866+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:30.558304+0000","last_became_peered":"2024-05-06T00:30:30.558304+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:58:28.298863+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.058990953999999998,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3578,"num_read_kb":12907,"num_write":404,"num_write_kb":3281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"199'918","reported_seq":9351,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667663+0000","last_change":"2024-05-06T00:30:39.656204+0000","last_active":"2024-05-06T00:30:40.667663+0000","last_peered":"2024-05-06T00:30:40.667663+0000","last_clean":"2024-05-06T00:30:40.667663+0000","last_became_active":"2024-05-06T00:30:39.653000+0000","last_became_peered":"2024-05-06T00:30:39.653000+0000","last_unstale":"2024-05-06T00:30:40.667663+0000","last_undegraded":"2024-05-06T00:30:40.667663+0000","last_fullsized":"2024-05-06T00:30:40.667663+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:11:01.551395+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.035914631000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":15,"num_read":12749,"num_read_kb":34289,"num_write":814,"num_write_kb":5275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"199'1204","reported_seq":10121,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725286+0000","last_change":"2024-05-06T00:30:30.704483+0000","last_active":"2024-05-06T00:30:40.725286+0000","last_peered":"2024-05-06T00:30:40.725286+0000","last_clean":"2024-05-06T00:30:40.725286+0000","last_became_active":"2024-05-06T00:30:30.568390+0000","last_became_peered":"2024-05-06T00:30:30.568390+0000","last_unstale":"2024-05-06T00:30:40.725286+0000","last_undegraded":"2024-05-06T00:30:40.725286+0000","last_fullsized":"2024-05-06T00:30:40.725286+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:14:14.323344+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.044642163999999998,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13696,"num_read_kb":33605,"num_write":1130,"num_write_kb":8108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"199'407","reported_seq":6998,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725360+0000","last_change":"2024-05-06T00:30:30.665158+0000","last_active":"2024-05-06T00:30:40.725360+0000","last_peered":"2024-05-06T00:30:40.725360+0000","last_clean":"2024-05-06T00:30:40.725360+0000","last_became_active":"2024-05-06T00:30:30.543930+0000","last_became_peered":"2024-05-06T00:30:30.543930+0000","last_unstale":"2024-05-06T00:30:40.725360+0000","last_undegraded":"2024-05-06T00:30:40.725360+0000","last_fullsized":"2024-05-06T00:30:40.725360+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:42:25.940056+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.040220385999999997,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":7280,"num_read_kb":25626,"num_write":414,"num_write_kb":13645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'681","reported_seq":3865,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668064+0000","last_change":"2024-05-06T00:30:30.639087+0000","last_active":"2024-05-06T00:30:40.668064+0000","last_peered":"2024-05-06T00:30:40.668064+0000","last_clean":"2024-05-06T00:30:40.668064+0000","last_became_active":"2024-05-06T00:30:30.560389+0000","last_became_peered":"2024-05-06T00:30:30.560389+0000","last_unstale":"2024-05-06T00:30:40.668064+0000","last_undegraded":"2024-05-06T00:30:40.668064+0000","last_fullsized":"2024-05-06T00:30:40.668064+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:47:13.586263+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.031279479999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5123,"num_read_kb":16860,"num_write":822,"num_write_kb":5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"185'748","reported_seq":14689,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725370+0000","last_change":"2024-05-06T00:30:35.598576+0000","last_active":"2024-05-06T00:30:40.725370+0000","last_peered":"2024-05-06T00:30:40.725370+0000","last_clean":"2024-05-06T00:30:40.725370+0000","last_became_active":"2024-05-06T00:30:35.595815+0000","last_became_peered":"2024-05-06T00:30:35.595815+0000","last_unstale":"2024-05-06T00:30:40.725370+0000","last_undegraded":"2024-05-06T00:30:40.725370+0000","last_fullsized":"2024-05-06T00:30:40.725370+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:45:37.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017909320000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":14404,"num_read_kb":28468,"num_write":531,"num_write_kb":3442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"194'349","reported_seq":4923,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725623+0000","last_change":"2024-05-06T00:30:19.957321+0000","last_active":"2024-05-06T00:30:40.725623+0000","last_peered":"2024-05-06T00:30:40.725623+0000","last_clean":"2024-05-06T00:30:40.725623+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:30:40.725623+0000","last_undegraded":"2024-05-06T00:30:40.725623+0000","last_fullsized":"2024-05-06T00:30:40.725623+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:27:18.137701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003171063,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3664,"num_read_kb":19738,"num_write":297,"num_write_kb":5164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"191'520","reported_seq":2900,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667838+0000","last_change":"2024-05-06T00:30:19.944378+0000","last_active":"2024-05-06T00:30:40.667838+0000","last_peered":"2024-05-06T00:30:40.667838+0000","last_clean":"2024-05-06T00:30:40.667838+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:30:40.667838+0000","last_undegraded":"2024-05-06T00:30:40.667838+0000","last_fullsized":"2024-05-06T00:30:40.667838+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:02:49.118813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004495626,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2722,"num_read_kb":15200,"num_write":616,"num_write_kb":1916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"191'298","reported_seq":5536,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665623+0000","last_change":"2024-05-06T00:30:36.646099+0000","last_active":"2024-05-06T00:30:40.665623+0000","last_peered":"2024-05-06T00:30:40.665623+0000","last_clean":"2024-05-06T00:30:40.665623+0000","last_became_active":"2024-05-06T00:30:36.641378+0000","last_became_peered":"2024-05-06T00:30:36.641378+0000","last_unstale":"2024-05-06T00:30:40.665623+0000","last_undegraded":"2024-05-06T00:30:40.665623+0000","last_fullsized":"2024-05-06T00:30:40.665623+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:03:14.811762+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031319316999999999,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":4880,"num_read_kb":11906,"num_write":264,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":76800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"199'529","reported_seq":7922,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665809+0000","last_change":"2024-05-06T00:30:36.655087+0000","last_active":"2024-05-06T00:30:40.665809+0000","last_peered":"2024-05-06T00:30:40.665809+0000","last_clean":"2024-05-06T00:30:40.665809+0000","last_became_active":"2024-05-06T00:30:36.651695+0000","last_became_peered":"2024-05-06T00:30:36.651695+0000","last_unstale":"2024-05-06T00:30:40.665809+0000","last_undegraded":"2024-05-06T00:30:40.665809+0000","last_fullsized":"2024-05-06T00:30:40.665809+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:30.279994+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.030926555000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":7962,"num_read_kb":31384,"num_write":673,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":237,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666855+0000","last_change":"2024-05-06T00:30:36.653124+0000","last_active":"2024-05-06T00:30:40.666855+0000","last_peered":"2024-05-06T00:30:40.666855+0000","last_clean":"2024-05-06T00:30:40.666855+0000","last_became_active":"2024-05-06T00:30:36.652331+0000","last_became_peered":"2024-05-06T00:30:36.652331+0000","last_unstale":"2024-05-06T00:30:40.666855+0000","last_undegraded":"2024-05-06T00:30:40.666855+0000","last_fullsized":"2024-05-06T00:30:40.666855+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:45:22.047908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"199'1005","reported_seq":7890,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725264+0000","last_change":"2024-05-06T00:30:30.709509+0000","last_active":"2024-05-06T00:30:40.725264+0000","last_peered":"2024-05-06T00:30:40.725264+0000","last_clean":"2024-05-06T00:30:40.725264+0000","last_became_active":"2024-05-06T00:30:30.569873+0000","last_became_peered":"2024-05-06T00:30:30.569873+0000","last_unstale":"2024-05-06T00:30:40.725264+0000","last_undegraded":"2024-05-06T00:30:40.725264+0000","last_fullsized":"2024-05-06T00:30:40.725264+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:37:31.390169+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.024654692999999998,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":10870,"num_read_kb":29913,"num_write":1138,"num_write_kb":7394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668489+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:30:40.668489+0000","last_peered":"2024-05-06T00:30:40.668489+0000","last_clean":"2024-05-06T00:30:40.668489+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:30:40.668489+0000","last_undegraded":"2024-05-06T00:30:40.668489+0000","last_fullsized":"2024-05-06T00:30:40.668489+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"191'766","reported_seq":6743,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725403+0000","last_change":"2024-05-06T00:30:35.593017+0000","last_active":"2024-05-06T00:30:40.725403+0000","last_peered":"2024-05-06T00:30:40.725403+0000","last_clean":"2024-05-06T00:30:40.725403+0000","last_became_active":"2024-05-06T00:30:35.589029+0000","last_became_peered":"2024-05-06T00:30:35.589029+0000","last_unstale":"2024-05-06T00:30:40.725403+0000","last_undegraded":"2024-05-06T00:30:40.725403+0000","last_fullsized":"2024-05-06T00:30:40.725403+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:18:10.559148+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.003902722,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8939,"num_read_kb":19825,"num_write":709,"num_write_kb":2787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666154+0000","last_change":"2024-05-06T00:30:36.646507+0000","last_active":"2024-05-06T00:30:40.666154+0000","last_peered":"2024-05-06T00:30:40.666154+0000","last_clean":"2024-05-06T00:30:40.666154+0000","last_became_active":"2024-05-06T00:30:36.644967+0000","last_became_peered":"2024-05-06T00:30:36.644967+0000","last_unstale":"2024-05-06T00:30:40.666154+0000","last_undegraded":"2024-05-06T00:30:40.666154+0000","last_fullsized":"2024-05-06T00:30:40.666154+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:16:47.890394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"191'657","reported_seq":9441,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668189+0000","last_change":"2024-05-06T00:30:19.945917+0000","last_active":"2024-05-06T00:30:40.668189+0000","last_peered":"2024-05-06T00:30:40.668189+0000","last_clean":"2024-05-06T00:30:40.668189+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:30:40.668189+0000","last_undegraded":"2024-05-06T00:30:40.668189+0000","last_fullsized":"2024-05-06T00:30:40.668189+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:49:12.946087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060800150000000002,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":10036,"num_read_kb":30901,"num_write":694,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668452+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:30:40.668452+0000","last_peered":"2024-05-06T00:30:40.668452+0000","last_clean":"2024-05-06T00:30:40.668452+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:30:40.668452+0000","last_undegraded":"2024-05-06T00:30:40.668452+0000","last_fullsized":"2024-05-06T00:30:40.668452+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"191'559","reported_seq":5683,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667793+0000","last_change":"2024-05-06T00:30:19.941179+0000","last_active":"2024-05-06T00:30:40.667793+0000","last_peered":"2024-05-06T00:30:40.667793+0000","last_clean":"2024-05-06T00:30:40.667793+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:30:40.667793+0000","last_undegraded":"2024-05-06T00:30:40.667793+0000","last_fullsized":"2024-05-06T00:30:40.667793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:38:10.377797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016637620000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":6637,"num_read_kb":16275,"num_write":486,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"199'538","reported_seq":4051,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725423+0000","last_change":"2024-05-06T00:30:30.742622+0000","last_active":"2024-05-06T00:30:40.725423+0000","last_peered":"2024-05-06T00:30:40.725423+0000","last_clean":"2024-05-06T00:30:40.725423+0000","last_became_active":"2024-05-06T00:30:30.561444+0000","last_became_peered":"2024-05-06T00:30:30.561444+0000","last_unstale":"2024-05-06T00:30:40.725423+0000","last_undegraded":"2024-05-06T00:30:40.725423+0000","last_fullsized":"2024-05-06T00:30:40.725423+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:39:30.619891+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.034165040000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":17,"num_read":3499,"num_read_kb":25511,"num_write":548,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":105,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667480+0000","last_change":"2024-05-06T00:30:39.648249+0000","last_active":"2024-05-06T00:30:40.667480+0000","last_peered":"2024-05-06T00:30:40.667480+0000","last_clean":"2024-05-06T00:30:40.667480+0000","last_became_active":"2024-05-06T00:30:39.647436+0000","last_became_peered":"2024-05-06T00:30:39.647436+0000","last_unstale":"2024-05-06T00:30:40.667480+0000","last_undegraded":"2024-05-06T00:30:40.667480+0000","last_fullsized":"2024-05-06T00:30:40.667480+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:15:38.102710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"192'657","reported_seq":6870,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667981+0000","last_change":"2024-05-06T00:30:19.944664+0000","last_active":"2024-05-06T00:30:40.667981+0000","last_peered":"2024-05-06T00:30:40.667981+0000","last_clean":"2024-05-06T00:30:40.667981+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:30:40.667981+0000","last_undegraded":"2024-05-06T00:30:40.667981+0000","last_fullsized":"2024-05-06T00:30:40.667981+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:56:10.880278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051286049999999996,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7148,"num_read_kb":27106,"num_write":776,"num_write_kb":4745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665403+0000","last_change":"2024-05-06T00:30:36.644291+0000","last_active":"2024-05-06T00:30:40.665403+0000","last_peered":"2024-05-06T00:30:40.665403+0000","last_clean":"2024-05-06T00:30:40.665403+0000","last_became_active":"2024-05-06T00:30:36.643456+0000","last_became_peered":"2024-05-06T00:30:36.643456+0000","last_unstale":"2024-05-06T00:30:40.665403+0000","last_undegraded":"2024-05-06T00:30:40.665403+0000","last_fullsized":"2024-05-06T00:30:40.665403+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:25:26.257376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"199'383","reported_seq":4994,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725003+0000","last_change":"2024-05-06T00:30:30.656302+0000","last_active":"2024-05-06T00:30:40.725003+0000","last_peered":"2024-05-06T00:30:40.725003+0000","last_clean":"2024-05-06T00:30:40.725003+0000","last_became_active":"2024-05-06T00:30:30.562288+0000","last_became_peered":"2024-05-06T00:30:30.562288+0000","last_unstale":"2024-05-06T00:30:40.725003+0000","last_undegraded":"2024-05-06T00:30:40.725003+0000","last_fullsized":"2024-05-06T00:30:40.725003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:53:00.473185+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.028356438000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":4034,"num_read_kb":14959,"num_write":372,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665364+0000","last_change":"2024-05-06T00:30:36.643966+0000","last_active":"2024-05-06T00:30:40.665364+0000","last_peered":"2024-05-06T00:30:40.665364+0000","last_clean":"2024-05-06T00:30:40.665364+0000","last_became_active":"2024-05-06T00:30:36.642148+0000","last_became_peered":"2024-05-06T00:30:36.642148+0000","last_unstale":"2024-05-06T00:30:40.665364+0000","last_undegraded":"2024-05-06T00:30:40.665364+0000","last_fullsized":"2024-05-06T00:30:40.665364+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:19:16.556193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"193'778","reported_seq":10467,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668186+0000","last_change":"2024-05-06T00:30:19.947559+0000","last_active":"2024-05-06T00:30:40.668186+0000","last_peered":"2024-05-06T00:30:40.668186+0000","last_clean":"2024-05-06T00:30:40.668186+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:30:40.668186+0000","last_undegraded":"2024-05-06T00:30:40.668186+0000","last_fullsized":"2024-05-06T00:30:40.668186+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:23:03.571190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065619830000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":12206,"num_read_kb":24770,"num_write":877,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667373+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:30:40.667373+0000","last_peered":"2024-05-06T00:30:40.667373+0000","last_clean":"2024-05-06T00:30:40.667373+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:30:40.667373+0000","last_undegraded":"2024-05-06T00:30:40.667373+0000","last_fullsized":"2024-05-06T00:30:40.667373+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":86,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665277+0000","last_change":"2024-05-06T00:30:36.642156+0000","last_active":"2024-05-06T00:30:40.665277+0000","last_peered":"2024-05-06T00:30:40.665277+0000","last_clean":"2024-05-06T00:30:40.665277+0000","last_became_active":"2024-05-06T00:30:36.641260+0000","last_became_peered":"2024-05-06T00:30:36.641260+0000","last_unstale":"2024-05-06T00:30:40.665277+0000","last_undegraded":"2024-05-06T00:30:40.665277+0000","last_fullsized":"2024-05-06T00:30:40.665277+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:31.624557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"191'620","reported_seq":6214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725308+0000","last_change":"2024-05-06T00:30:35.595715+0000","last_active":"2024-05-06T00:30:40.725308+0000","last_peered":"2024-05-06T00:30:40.725308+0000","last_clean":"2024-05-06T00:30:40.725308+0000","last_became_active":"2024-05-06T00:30:35.591860+0000","last_became_peered":"2024-05-06T00:30:35.591860+0000","last_unstale":"2024-05-06T00:30:40.725308+0000","last_undegraded":"2024-05-06T00:30:40.725308+0000","last_fullsized":"2024-05-06T00:30:40.725308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:47:33.295174+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.021797200999999999,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":6539,"num_read_kb":27944,"num_write":612,"num_write_kb":8155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"199'575","reported_seq":5828,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725392+0000","last_change":"2024-05-06T00:30:30.675754+0000","last_active":"2024-05-06T00:30:40.725392+0000","last_peered":"2024-05-06T00:30:40.725392+0000","last_clean":"2024-05-06T00:30:40.725392+0000","last_became_active":"2024-05-06T00:30:30.545607+0000","last_became_peered":"2024-05-06T00:30:30.545607+0000","last_unstale":"2024-05-06T00:30:40.725392+0000","last_undegraded":"2024-05-06T00:30:40.725392+0000","last_fullsized":"2024-05-06T00:30:40.725392+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:54:11.975131+0000","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.036169352000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":5497,"num_read_kb":27780,"num_write":687,"num_write_kb":13301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"199'805","reported_seq":8521,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665648+0000","last_change":"2024-05-06T00:30:36.656283+0000","last_active":"2024-05-06T00:30:40.665648+0000","last_peered":"2024-05-06T00:30:40.665648+0000","last_clean":"2024-05-06T00:30:40.665648+0000","last_became_active":"2024-05-06T00:30:36.651797+0000","last_became_peered":"2024-05-06T00:30:36.651797+0000","last_unstale":"2024-05-06T00:30:40.665648+0000","last_undegraded":"2024-05-06T00:30:40.665648+0000","last_fullsized":"2024-05-06T00:30:40.665648+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:47:48.879526+0000","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0.028632154999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":25,"num_read":9542,"num_read_kb":44861,"num_write":766,"num_write_kb":8347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"187'351","reported_seq":7154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668166+0000","last_change":"2024-05-06T00:30:19.946118+0000","last_active":"2024-05-06T00:30:40.668166+0000","last_peered":"2024-05-06T00:30:40.668166+0000","last_clean":"2024-05-06T00:30:40.668166+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:30:40.668166+0000","last_undegraded":"2024-05-06T00:30:40.668166+0000","last_fullsized":"2024-05-06T00:30:40.668166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:06:25.304781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057155160000000003,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5046,"num_read_kb":27757,"num_write":349,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"188'420","reported_seq":6843,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:39.652878+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:39.650059+0000","last_became_peered":"2024-05-06T00:30:39.650059+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:14:23.610386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057267869999999997,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":6116,"num_read_kb":54371,"num_write":370,"num_write_kb":8906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"187'890","reported_seq":12713,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666252+0000","last_change":"2024-05-06T00:30:36.650116+0000","last_active":"2024-05-06T00:30:40.666252+0000","last_peered":"2024-05-06T00:30:40.666252+0000","last_clean":"2024-05-06T00:30:40.666252+0000","last_became_active":"2024-05-06T00:30:36.645606+0000","last_became_peered":"2024-05-06T00:30:36.645606+0000","last_unstale":"2024-05-06T00:30:40.666252+0000","last_undegraded":"2024-05-06T00:30:40.666252+0000","last_fullsized":"2024-05-06T00:30:40.666252+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:34:38.495321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015459429999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":13504,"num_read_kb":33376,"num_write":1160,"num_write_kb":4653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"199'867","reported_seq":12480,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667893+0000","last_change":"2024-05-06T00:30:39.656310+0000","last_active":"2024-05-06T00:30:40.667893+0000","last_peered":"2024-05-06T00:30:40.667893+0000","last_clean":"2024-05-06T00:30:40.667893+0000","last_became_active":"2024-05-06T00:30:39.652991+0000","last_became_peered":"2024-05-06T00:30:39.652991+0000","last_unstale":"2024-05-06T00:30:40.667893+0000","last_undegraded":"2024-05-06T00:30:40.667893+0000","last_fullsized":"2024-05-06T00:30:40.667893+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:35:14.638153+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.035058351000000001,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":15156,"num_read_kb":31725,"num_write":1055,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"190'348","reported_seq":4258,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668084+0000","last_change":"2024-05-06T00:30:19.943335+0000","last_active":"2024-05-06T00:30:40.668084+0000","last_peered":"2024-05-06T00:30:40.668084+0000","last_clean":"2024-05-06T00:30:40.668084+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:30:40.668084+0000","last_undegraded":"2024-05-06T00:30:40.668084+0000","last_fullsized":"2024-05-06T00:30:40.668084+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:42:14.046101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033172480000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":3496,"num_read_kb":19343,"num_write":377,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'790","reported_seq":11712,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667348+0000","last_change":"2024-05-06T00:30:39.652246+0000","last_active":"2024-05-06T00:30:40.667348+0000","last_peered":"2024-05-06T00:30:40.667348+0000","last_clean":"2024-05-06T00:30:40.667348+0000","last_became_active":"2024-05-06T00:30:39.649412+0000","last_became_peered":"2024-05-06T00:30:39.649412+0000","last_unstale":"2024-05-06T00:30:40.667348+0000","last_undegraded":"2024-05-06T00:30:40.667348+0000","last_fullsized":"2024-05-06T00:30:40.667348+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:51:30.132753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001791927,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13318,"num_read_kb":32027,"num_write":733,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"199'437","reported_seq":5748,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665489+0000","last_change":"2024-05-06T00:30:36.655967+0000","last_active":"2024-05-06T00:30:40.665489+0000","last_peered":"2024-05-06T00:30:40.665489+0000","last_clean":"2024-05-06T00:30:40.665489+0000","last_became_active":"2024-05-06T00:30:36.652383+0000","last_became_peered":"2024-05-06T00:30:36.652383+0000","last_unstale":"2024-05-06T00:30:40.665489+0000","last_undegraded":"2024-05-06T00:30:40.665489+0000","last_fullsized":"2024-05-06T00:30:40.665489+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:56:34.123681+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.029993888999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5408,"num_read_kb":10751,"num_write":542,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"191'477","reported_seq":9126,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725278+0000","last_change":"2024-05-06T00:30:19.956297+0000","last_active":"2024-05-06T00:30:40.725278+0000","last_peered":"2024-05-06T00:30:40.725278+0000","last_clean":"2024-05-06T00:30:40.725278+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:30:40.725278+0000","last_undegraded":"2024-05-06T00:30:40.725278+0000","last_fullsized":"2024-05-06T00:30:40.725278+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:46.269848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002988648,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8695,"num_read_kb":23092,"num_write":389,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"192'673","reported_seq":9486,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665601+0000","last_change":"2024-05-06T00:30:36.651926+0000","last_active":"2024-05-06T00:30:40.665601+0000","last_peered":"2024-05-06T00:30:40.665601+0000","last_clean":"2024-05-06T00:30:40.665601+0000","last_became_active":"2024-05-06T00:30:36.647614+0000","last_became_peered":"2024-05-06T00:30:36.647614+0000","last_unstale":"2024-05-06T00:30:40.665601+0000","last_undegraded":"2024-05-06T00:30:40.665601+0000","last_fullsized":"2024-05-06T00:30:40.665601+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:16:00.581426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00351067,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":10781,"num_read_kb":40700,"num_write":619,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":623,"num_object_clones":0,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":619,"num_read":263747,"num_read_kb":883054,"num_write":20048,"num_write_kb":192626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5734,"num_bytes_recovered":72015234,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},"log_size":19745,"ondisk_log_size":19745,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640312,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60016,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311824,"statfs":{"total":96636764160,"available":96575307776,"internally_reserved":0,"allocated":2519040,"data_stored":1062321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":198,"seq":850403524612,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":2476,"kb_used_omap":44,"kb_used_meta":29331,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45664,"internal_metadata":30035360},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640310,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640311,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47544,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324296,"statfs":{"total":96636764160,"available":96588079104,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640312,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47552,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324288,"statfs":{"total":96636764160,"available":96588070912,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640310,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48116,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323724,"statfs":{"total":96636764160,"available":96587493376,"internally_reserved":0,"allocated":3112960,"data_stored":1652689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:30:43.831 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:30:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:44.106 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:30:44.123 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-05-06T00:30:42.460500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":625,"num_object_clones":0,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":619,"num_read":263813,"num_read_kb":883109,"num_write":20149,"num_write_kb":194254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5740,"num_bytes_recovered":73786338,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19817,"ondisk_log_size":19817,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":283216,"kb_used_data":15976,"kb_used_omap":52,"kb_used_meta":267083,"kb_avail":565947824,"statfs":{"total":579820584960,"available":579530571776,"internally_reserved":0,"allocated":16359424,"data_stored":7574696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53637,"internal_metadata":273493627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.824385"},"pg_stats":[{"pgid":"3.1f","version":"194'672","reported_seq":13023,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725510+0000","last_change":"2024-05-06T00:30:19.956846+0000","last_active":"2024-05-06T00:30:40.725510+0000","last_peered":"2024-05-06T00:30:40.725510+0000","last_clean":"2024-05-06T00:30:40.725510+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:30:40.725510+0000","last_undegraded":"2024-05-06T00:30:40.725510+0000","last_fullsized":"2024-05-06T00:30:40.725510+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:12:04.053936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028964080000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":12460,"num_read_kb":40065,"num_write":528,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"191'367","reported_seq":9956,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725314+0000","last_change":"2024-05-06T00:30:19.973477+0000","last_active":"2024-05-06T00:30:40.725314+0000","last_peered":"2024-05-06T00:30:40.725314+0000","last_clean":"2024-05-06T00:30:40.725314+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:30:40.725314+0000","last_undegraded":"2024-05-06T00:30:40.725314+0000","last_fullsized":"2024-05-06T00:30:40.725314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:17.412771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012177069999999999,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":8762,"num_read_kb":50019,"num_write":296,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"199'456","reported_seq":4006,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:30.637866+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:30.558304+0000","last_became_peered":"2024-05-06T00:30:30.558304+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:58:28.298863+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.058990953999999998,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3578,"num_read_kb":12907,"num_write":404,"num_write_kb":3281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"199'918","reported_seq":9351,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667663+0000","last_change":"2024-05-06T00:30:39.656204+0000","last_active":"2024-05-06T00:30:40.667663+0000","last_peered":"2024-05-06T00:30:40.667663+0000","last_clean":"2024-05-06T00:30:40.667663+0000","last_became_active":"2024-05-06T00:30:39.653000+0000","last_became_peered":"2024-05-06T00:30:39.653000+0000","last_unstale":"2024-05-06T00:30:40.667663+0000","last_undegraded":"2024-05-06T00:30:40.667663+0000","last_fullsized":"2024-05-06T00:30:40.667663+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:11:01.551395+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.035914631000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":15,"num_read":12749,"num_read_kb":34289,"num_write":814,"num_write_kb":5275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"199'1204","reported_seq":10121,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725286+0000","last_change":"2024-05-06T00:30:30.704483+0000","last_active":"2024-05-06T00:30:40.725286+0000","last_peered":"2024-05-06T00:30:40.725286+0000","last_clean":"2024-05-06T00:30:40.725286+0000","last_became_active":"2024-05-06T00:30:30.568390+0000","last_became_peered":"2024-05-06T00:30:30.568390+0000","last_unstale":"2024-05-06T00:30:40.725286+0000","last_undegraded":"2024-05-06T00:30:40.725286+0000","last_fullsized":"2024-05-06T00:30:40.725286+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:14:14.323344+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.044642163999999998,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13696,"num_read_kb":33605,"num_write":1130,"num_write_kb":8108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"199'407","reported_seq":6998,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725360+0000","last_change":"2024-05-06T00:30:30.665158+0000","last_active":"2024-05-06T00:30:40.725360+0000","last_peered":"2024-05-06T00:30:40.725360+0000","last_clean":"2024-05-06T00:30:40.725360+0000","last_became_active":"2024-05-06T00:30:30.543930+0000","last_became_peered":"2024-05-06T00:30:30.543930+0000","last_unstale":"2024-05-06T00:30:40.725360+0000","last_undegraded":"2024-05-06T00:30:40.725360+0000","last_fullsized":"2024-05-06T00:30:40.725360+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:42:25.940056+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.040220385999999997,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":7280,"num_read_kb":25626,"num_write":414,"num_write_kb":13645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'681","reported_seq":3865,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668064+0000","last_change":"2024-05-06T00:30:30.639087+0000","last_active":"2024-05-06T00:30:40.668064+0000","last_peered":"2024-05-06T00:30:40.668064+0000","last_clean":"2024-05-06T00:30:40.668064+0000","last_became_active":"2024-05-06T00:30:30.560389+0000","last_became_peered":"2024-05-06T00:30:30.560389+0000","last_unstale":"2024-05-06T00:30:40.668064+0000","last_undegraded":"2024-05-06T00:30:40.668064+0000","last_fullsized":"2024-05-06T00:30:40.668064+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:47:13.586263+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.031279479999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5123,"num_read_kb":16860,"num_write":822,"num_write_kb":5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"185'748","reported_seq":14689,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725370+0000","last_change":"2024-05-06T00:30:35.598576+0000","last_active":"2024-05-06T00:30:40.725370+0000","last_peered":"2024-05-06T00:30:40.725370+0000","last_clean":"2024-05-06T00:30:40.725370+0000","last_became_active":"2024-05-06T00:30:35.595815+0000","last_became_peered":"2024-05-06T00:30:35.595815+0000","last_unstale":"2024-05-06T00:30:40.725370+0000","last_undegraded":"2024-05-06T00:30:40.725370+0000","last_fullsized":"2024-05-06T00:30:40.725370+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:45:37.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017909320000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":14404,"num_read_kb":28468,"num_write":531,"num_write_kb":3442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"194'349","reported_seq":4923,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725623+0000","last_change":"2024-05-06T00:30:19.957321+0000","last_active":"2024-05-06T00:30:40.725623+0000","last_peered":"2024-05-06T00:30:40.725623+0000","last_clean":"2024-05-06T00:30:40.725623+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:30:40.725623+0000","last_undegraded":"2024-05-06T00:30:40.725623+0000","last_fullsized":"2024-05-06T00:30:40.725623+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:27:18.137701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003171063,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3664,"num_read_kb":19738,"num_write":297,"num_write_kb":5164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"191'520","reported_seq":2900,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667838+0000","last_change":"2024-05-06T00:30:19.944378+0000","last_active":"2024-05-06T00:30:40.667838+0000","last_peered":"2024-05-06T00:30:40.667838+0000","last_clean":"2024-05-06T00:30:40.667838+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:30:40.667838+0000","last_undegraded":"2024-05-06T00:30:40.667838+0000","last_fullsized":"2024-05-06T00:30:40.667838+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:02:49.118813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004495626,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2722,"num_read_kb":15200,"num_write":616,"num_write_kb":1916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"191'298","reported_seq":5536,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665623+0000","last_change":"2024-05-06T00:30:36.646099+0000","last_active":"2024-05-06T00:30:40.665623+0000","last_peered":"2024-05-06T00:30:40.665623+0000","last_clean":"2024-05-06T00:30:40.665623+0000","last_became_active":"2024-05-06T00:30:36.641378+0000","last_became_peered":"2024-05-06T00:30:36.641378+0000","last_unstale":"2024-05-06T00:30:40.665623+0000","last_undegraded":"2024-05-06T00:30:40.665623+0000","last_fullsized":"2024-05-06T00:30:40.665623+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:03:14.811762+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031319316999999999,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":4880,"num_read_kb":11906,"num_write":264,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":76800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"199'529","reported_seq":7922,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665809+0000","last_change":"2024-05-06T00:30:36.655087+0000","last_active":"2024-05-06T00:30:40.665809+0000","last_peered":"2024-05-06T00:30:40.665809+0000","last_clean":"2024-05-06T00:30:40.665809+0000","last_became_active":"2024-05-06T00:30:36.651695+0000","last_became_peered":"2024-05-06T00:30:36.651695+0000","last_unstale":"2024-05-06T00:30:40.665809+0000","last_undegraded":"2024-05-06T00:30:40.665809+0000","last_fullsized":"2024-05-06T00:30:40.665809+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:30.279994+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.030926555000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":7962,"num_read_kb":31384,"num_write":673,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":237,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666855+0000","last_change":"2024-05-06T00:30:36.653124+0000","last_active":"2024-05-06T00:30:40.666855+0000","last_peered":"2024-05-06T00:30:40.666855+0000","last_clean":"2024-05-06T00:30:40.666855+0000","last_became_active":"2024-05-06T00:30:36.652331+0000","last_became_peered":"2024-05-06T00:30:36.652331+0000","last_unstale":"2024-05-06T00:30:40.666855+0000","last_undegraded":"2024-05-06T00:30:40.666855+0000","last_fullsized":"2024-05-06T00:30:40.666855+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:45:22.047908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"199'1005","reported_seq":7890,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725264+0000","last_change":"2024-05-06T00:30:30.709509+0000","last_active":"2024-05-06T00:30:40.725264+0000","last_peered":"2024-05-06T00:30:40.725264+0000","last_clean":"2024-05-06T00:30:40.725264+0000","last_became_active":"2024-05-06T00:30:30.569873+0000","last_became_peered":"2024-05-06T00:30:30.569873+0000","last_unstale":"2024-05-06T00:30:40.725264+0000","last_undegraded":"2024-05-06T00:30:40.725264+0000","last_fullsized":"2024-05-06T00:30:40.725264+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:37:31.390169+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.024654692999999998,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":10870,"num_read_kb":29913,"num_write":1138,"num_write_kb":7394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668489+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:30:40.668489+0000","last_peered":"2024-05-06T00:30:40.668489+0000","last_clean":"2024-05-06T00:30:40.668489+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:30:40.668489+0000","last_undegraded":"2024-05-06T00:30:40.668489+0000","last_fullsized":"2024-05-06T00:30:40.668489+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"191'766","reported_seq":6743,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725403+0000","last_change":"2024-05-06T00:30:35.593017+0000","last_active":"2024-05-06T00:30:40.725403+0000","last_peered":"2024-05-06T00:30:40.725403+0000","last_clean":"2024-05-06T00:30:40.725403+0000","last_became_active":"2024-05-06T00:30:35.589029+0000","last_became_peered":"2024-05-06T00:30:35.589029+0000","last_unstale":"2024-05-06T00:30:40.725403+0000","last_undegraded":"2024-05-06T00:30:40.725403+0000","last_fullsized":"2024-05-06T00:30:40.725403+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:18:10.559148+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.003902722,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8939,"num_read_kb":19825,"num_write":709,"num_write_kb":2787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666154+0000","last_change":"2024-05-06T00:30:36.646507+0000","last_active":"2024-05-06T00:30:40.666154+0000","last_peered":"2024-05-06T00:30:40.666154+0000","last_clean":"2024-05-06T00:30:40.666154+0000","last_became_active":"2024-05-06T00:30:36.644967+0000","last_became_peered":"2024-05-06T00:30:36.644967+0000","last_unstale":"2024-05-06T00:30:40.666154+0000","last_undegraded":"2024-05-06T00:30:40.666154+0000","last_fullsized":"2024-05-06T00:30:40.666154+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:16:47.890394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"191'657","reported_seq":9441,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668189+0000","last_change":"2024-05-06T00:30:19.945917+0000","last_active":"2024-05-06T00:30:40.668189+0000","last_peered":"2024-05-06T00:30:40.668189+0000","last_clean":"2024-05-06T00:30:40.668189+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:30:40.668189+0000","last_undegraded":"2024-05-06T00:30:40.668189+0000","last_fullsized":"2024-05-06T00:30:40.668189+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:49:12.946087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060800150000000002,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":10036,"num_read_kb":30901,"num_write":694,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668452+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:30:40.668452+0000","last_peered":"2024-05-06T00:30:40.668452+0000","last_clean":"2024-05-06T00:30:40.668452+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:30:40.668452+0000","last_undegraded":"2024-05-06T00:30:40.668452+0000","last_fullsized":"2024-05-06T00:30:40.668452+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"191'559","reported_seq":5683,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667793+0000","last_change":"2024-05-06T00:30:19.941179+0000","last_active":"2024-05-06T00:30:40.667793+0000","last_peered":"2024-05-06T00:30:40.667793+0000","last_clean":"2024-05-06T00:30:40.667793+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:30:40.667793+0000","last_undegraded":"2024-05-06T00:30:40.667793+0000","last_fullsized":"2024-05-06T00:30:40.667793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:38:10.377797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016637620000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":6637,"num_read_kb":16275,"num_write":486,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"199'538","reported_seq":4051,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725423+0000","last_change":"2024-05-06T00:30:30.742622+0000","last_active":"2024-05-06T00:30:40.725423+0000","last_peered":"2024-05-06T00:30:40.725423+0000","last_clean":"2024-05-06T00:30:40.725423+0000","last_became_active":"2024-05-06T00:30:30.561444+0000","last_became_peered":"2024-05-06T00:30:30.561444+0000","last_unstale":"2024-05-06T00:30:40.725423+0000","last_undegraded":"2024-05-06T00:30:40.725423+0000","last_fullsized":"2024-05-06T00:30:40.725423+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:39:30.619891+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.034165040000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":17,"num_read":3499,"num_read_kb":25511,"num_write":548,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":105,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667480+0000","last_change":"2024-05-06T00:30:39.648249+0000","last_active":"2024-05-06T00:30:40.667480+0000","last_peered":"2024-05-06T00:30:40.667480+0000","last_clean":"2024-05-06T00:30:40.667480+0000","last_became_active":"2024-05-06T00:30:39.647436+0000","last_became_peered":"2024-05-06T00:30:39.647436+0000","last_unstale":"2024-05-06T00:30:40.667480+0000","last_undegraded":"2024-05-06T00:30:40.667480+0000","last_fullsized":"2024-05-06T00:30:40.667480+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:15:38.102710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"192'657","reported_seq":6870,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667981+0000","last_change":"2024-05-06T00:30:19.944664+0000","last_active":"2024-05-06T00:30:40.667981+0000","last_peered":"2024-05-06T00:30:40.667981+0000","last_clean":"2024-05-06T00:30:40.667981+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:30:40.667981+0000","last_undegraded":"2024-05-06T00:30:40.667981+0000","last_fullsized":"2024-05-06T00:30:40.667981+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:56:10.880278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051286049999999996,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7148,"num_read_kb":27106,"num_write":776,"num_write_kb":4745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665403+0000","last_change":"2024-05-06T00:30:36.644291+0000","last_active":"2024-05-06T00:30:40.665403+0000","last_peered":"2024-05-06T00:30:40.665403+0000","last_clean":"2024-05-06T00:30:40.665403+0000","last_became_active":"2024-05-06T00:30:36.643456+0000","last_became_peered":"2024-05-06T00:30:36.643456+0000","last_unstale":"2024-05-06T00:30:40.665403+0000","last_undegraded":"2024-05-06T00:30:40.665403+0000","last_fullsized":"2024-05-06T00:30:40.665403+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:25:26.257376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"199'383","reported_seq":4994,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725003+0000","last_change":"2024-05-06T00:30:30.656302+0000","last_active":"2024-05-06T00:30:40.725003+0000","last_peered":"2024-05-06T00:30:40.725003+0000","last_clean":"2024-05-06T00:30:40.725003+0000","last_became_active":"2024-05-06T00:30:30.562288+0000","last_became_peered":"2024-05-06T00:30:30.562288+0000","last_unstale":"2024-05-06T00:30:40.725003+0000","last_undegraded":"2024-05-06T00:30:40.725003+0000","last_fullsized":"2024-05-06T00:30:40.725003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:53:00.473185+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.028356438000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":4034,"num_read_kb":14959,"num_write":372,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665364+0000","last_change":"2024-05-06T00:30:36.643966+0000","last_active":"2024-05-06T00:30:40.665364+0000","last_peered":"2024-05-06T00:30:40.665364+0000","last_clean":"2024-05-06T00:30:40.665364+0000","last_became_active":"2024-05-06T00:30:36.642148+0000","last_became_peered":"2024-05-06T00:30:36.642148+0000","last_unstale":"2024-05-06T00:30:40.665364+0000","last_undegraded":"2024-05-06T00:30:40.665364+0000","last_fullsized":"2024-05-06T00:30:40.665364+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:19:16.556193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"193'778","reported_seq":10467,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668186+0000","last_change":"2024-05-06T00:30:19.947559+0000","last_active":"2024-05-06T00:30:40.668186+0000","last_peered":"2024-05-06T00:30:40.668186+0000","last_clean":"2024-05-06T00:30:40.668186+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:30:40.668186+0000","last_undegraded":"2024-05-06T00:30:40.668186+0000","last_fullsized":"2024-05-06T00:30:40.668186+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:23:03.571190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065619830000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":12206,"num_read_kb":24770,"num_write":877,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667373+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:30:40.667373+0000","last_peered":"2024-05-06T00:30:40.667373+0000","last_clean":"2024-05-06T00:30:40.667373+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:30:40.667373+0000","last_undegraded":"2024-05-06T00:30:40.667373+0000","last_fullsized":"2024-05-06T00:30:40.667373+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":86,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665277+0000","last_change":"2024-05-06T00:30:36.642156+0000","last_active":"2024-05-06T00:30:40.665277+0000","last_peered":"2024-05-06T00:30:40.665277+0000","last_clean":"2024-05-06T00:30:40.665277+0000","last_became_active":"2024-05-06T00:30:36.641260+0000","last_became_peered":"2024-05-06T00:30:36.641260+0000","last_unstale":"2024-05-06T00:30:40.665277+0000","last_undegraded":"2024-05-06T00:30:40.665277+0000","last_fullsized":"2024-05-06T00:30:40.665277+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:31.624557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"191'620","reported_seq":6214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725308+0000","last_change":"2024-05-06T00:30:35.595715+0000","last_active":"2024-05-06T00:30:40.725308+0000","last_peered":"2024-05-06T00:30:40.725308+0000","last_clean":"2024-05-06T00:30:40.725308+0000","last_became_active":"2024-05-06T00:30:35.591860+0000","last_became_peered":"2024-05-06T00:30:35.591860+0000","last_unstale":"2024-05-06T00:30:40.725308+0000","last_undegraded":"2024-05-06T00:30:40.725308+0000","last_fullsized":"2024-05-06T00:30:40.725308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:47:33.295174+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.021797200999999999,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":6539,"num_read_kb":27944,"num_write":612,"num_write_kb":8155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"199'575","reported_seq":5828,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725392+0000","last_change":"2024-05-06T00:30:30.675754+0000","last_active":"2024-05-06T00:30:40.725392+0000","last_peered":"2024-05-06T00:30:40.725392+0000","last_clean":"2024-05-06T00:30:40.725392+0000","last_became_active":"2024-05-06T00:30:30.545607+0000","last_became_peered":"2024-05-06T00:30:30.545607+0000","last_unstale":"2024-05-06T00:30:40.725392+0000","last_undegraded":"2024-05-06T00:30:40.725392+0000","last_fullsized":"2024-05-06T00:30:40.725392+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:54:11.975131+0000","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.036169352000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":5497,"num_read_kb":27780,"num_write":687,"num_write_kb":13301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"199'805","reported_seq":8521,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665648+0000","last_change":"2024-05-06T00:30:36.656283+0000","last_active":"2024-05-06T00:30:40.665648+0000","last_peered":"2024-05-06T00:30:40.665648+0000","last_clean":"2024-05-06T00:30:40.665648+0000","last_became_active":"2024-05-06T00:30:36.651797+0000","last_became_peered":"2024-05-06T00:30:36.651797+0000","last_unstale":"2024-05-06T00:30:40.665648+0000","last_undegraded":"2024-05-06T00:30:40.665648+0000","last_fullsized":"2024-05-06T00:30:40.665648+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:47:48.879526+0000","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0.028632154999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":25,"num_read":9542,"num_read_kb":44861,"num_write":766,"num_write_kb":8347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"187'351","reported_seq":7154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668166+0000","last_change":"2024-05-06T00:30:19.946118+0000","last_active":"2024-05-06T00:30:40.668166+0000","last_peered":"2024-05-06T00:30:40.668166+0000","last_clean":"2024-05-06T00:30:40.668166+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:30:40.668166+0000","last_undegraded":"2024-05-06T00:30:40.668166+0000","last_fullsized":"2024-05-06T00:30:40.668166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:06:25.304781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057155160000000003,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5046,"num_read_kb":27757,"num_write":349,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"188'420","reported_seq":6843,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:39.652878+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:39.650059+0000","last_became_peered":"2024-05-06T00:30:39.650059+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:14:23.610386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057267869999999997,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":6116,"num_read_kb":54371,"num_write":370,"num_write_kb":8906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"187'890","reported_seq":12713,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666252+0000","last_change":"2024-05-06T00:30:36.650116+0000","last_active":"2024-05-06T00:30:40.666252+0000","last_peered":"2024-05-06T00:30:40.666252+0000","last_clean":"2024-05-06T00:30:40.666252+0000","last_became_active":"2024-05-06T00:30:36.645606+0000","last_became_peered":"2024-05-06T00:30:36.645606+0000","last_unstale":"2024-05-06T00:30:40.666252+0000","last_undegraded":"2024-05-06T00:30:40.666252+0000","last_fullsized":"2024-05-06T00:30:40.666252+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:34:38.495321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015459429999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":13504,"num_read_kb":33376,"num_write":1160,"num_write_kb":4653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"199'867","reported_seq":12480,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667893+0000","last_change":"2024-05-06T00:30:39.656310+0000","last_active":"2024-05-06T00:30:40.667893+0000","last_peered":"2024-05-06T00:30:40.667893+0000","last_clean":"2024-05-06T00:30:40.667893+0000","last_became_active":"2024-05-06T00:30:39.652991+0000","last_became_peered":"2024-05-06T00:30:39.652991+0000","last_unstale":"2024-05-06T00:30:40.667893+0000","last_undegraded":"2024-05-06T00:30:40.667893+0000","last_fullsized":"2024-05-06T00:30:40.667893+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:35:14.638153+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.035058351000000001,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":15156,"num_read_kb":31725,"num_write":1055,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"190'348","reported_seq":4258,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668084+0000","last_change":"2024-05-06T00:30:19.943335+0000","last_active":"2024-05-06T00:30:40.668084+0000","last_peered":"2024-05-06T00:30:40.668084+0000","last_clean":"2024-05-06T00:30:40.668084+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:30:40.668084+0000","last_undegraded":"2024-05-06T00:30:40.668084+0000","last_fullsized":"2024-05-06T00:30:40.668084+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:42:14.046101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033172480000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":3496,"num_read_kb":19343,"num_write":377,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'790","reported_seq":11712,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667348+0000","last_change":"2024-05-06T00:30:39.652246+0000","last_active":"2024-05-06T00:30:40.667348+0000","last_peered":"2024-05-06T00:30:40.667348+0000","last_clean":"2024-05-06T00:30:40.667348+0000","last_became_active":"2024-05-06T00:30:39.649412+0000","last_became_peered":"2024-05-06T00:30:39.649412+0000","last_unstale":"2024-05-06T00:30:40.667348+0000","last_undegraded":"2024-05-06T00:30:40.667348+0000","last_fullsized":"2024-05-06T00:30:40.667348+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:51:30.132753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001791927,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13318,"num_read_kb":32027,"num_write":733,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"199'437","reported_seq":5748,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665489+0000","last_change":"2024-05-06T00:30:36.655967+0000","last_active":"2024-05-06T00:30:40.665489+0000","last_peered":"2024-05-06T00:30:40.665489+0000","last_clean":"2024-05-06T00:30:40.665489+0000","last_became_active":"2024-05-06T00:30:36.652383+0000","last_became_peered":"2024-05-06T00:30:36.652383+0000","last_unstale":"2024-05-06T00:30:40.665489+0000","last_undegraded":"2024-05-06T00:30:40.665489+0000","last_fullsized":"2024-05-06T00:30:40.665489+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:56:34.123681+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.029993888999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5408,"num_read_kb":10751,"num_write":542,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"191'477","reported_seq":9126,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725278+0000","last_change":"2024-05-06T00:30:19.956297+0000","last_active":"2024-05-06T00:30:40.725278+0000","last_peered":"2024-05-06T00:30:40.725278+0000","last_clean":"2024-05-06T00:30:40.725278+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:30:40.725278+0000","last_undegraded":"2024-05-06T00:30:40.725278+0000","last_fullsized":"2024-05-06T00:30:40.725278+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:46.269848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002988648,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8695,"num_read_kb":23092,"num_write":389,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"192'673","reported_seq":9486,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665601+0000","last_change":"2024-05-06T00:30:36.651926+0000","last_active":"2024-05-06T00:30:40.665601+0000","last_peered":"2024-05-06T00:30:40.665601+0000","last_clean":"2024-05-06T00:30:40.665601+0000","last_became_active":"2024-05-06T00:30:36.647614+0000","last_became_peered":"2024-05-06T00:30:36.647614+0000","last_unstale":"2024-05-06T00:30:40.665601+0000","last_undegraded":"2024-05-06T00:30:40.665601+0000","last_fullsized":"2024-05-06T00:30:40.665601+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:16:00.581426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00351067,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":10781,"num_read_kb":40700,"num_write":619,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":623,"num_object_clones":0,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":619,"num_read":263747,"num_read_kb":883054,"num_write":20048,"num_write_kb":192626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5734,"num_bytes_recovered":72015234,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},"log_size":19745,"ondisk_log_size":19745,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640312,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60016,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311824,"statfs":{"total":96636764160,"available":96575307776,"internally_reserved":0,"allocated":2519040,"data_stored":1062321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":198,"seq":850403524612,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":2476,"kb_used_omap":44,"kb_used_meta":29331,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45664,"internal_metadata":30035360},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640310,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640311,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47544,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324296,"statfs":{"total":96636764160,"available":96588079104,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640312,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47552,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324288,"statfs":{"total":96636764160,"available":96588070912,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640310,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48116,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323724,"statfs":{"total":96636764160,"available":96587493376,"internally_reserved":0,"allocated":3112960,"data_stored":1652689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:30:44.125 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-05-06T00:30:44.125 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-05-06T00:30:44.416 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-06T00:30:44.426 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-06T00:30:44.426 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:30:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:44.710 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:30:44.727 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-05-06T00:30:44.461212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":625,"num_object_clones":0,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":619,"num_read":263813,"num_read_kb":883109,"num_write":20149,"num_write_kb":194254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5740,"num_bytes_recovered":73786338,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19817,"ondisk_log_size":19817,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":283248,"kb_used_data":16000,"kb_used_omap":52,"kb_used_meta":267083,"kb_avail":565947792,"statfs":{"total":579820584960,"available":579530539008,"internally_reserved":0,"allocated":16384000,"data_stored":7584694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53637,"internal_metadata":273493627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001783"},"pg_stats":[{"pgid":"3.1f","version":"194'672","reported_seq":13023,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725510+0000","last_change":"2024-05-06T00:30:19.956846+0000","last_active":"2024-05-06T00:30:40.725510+0000","last_peered":"2024-05-06T00:30:40.725510+0000","last_clean":"2024-05-06T00:30:40.725510+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:30:40.725510+0000","last_undegraded":"2024-05-06T00:30:40.725510+0000","last_fullsized":"2024-05-06T00:30:40.725510+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:12:04.053936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028964080000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":12460,"num_read_kb":40065,"num_write":528,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"191'367","reported_seq":9956,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725314+0000","last_change":"2024-05-06T00:30:19.973477+0000","last_active":"2024-05-06T00:30:40.725314+0000","last_peered":"2024-05-06T00:30:40.725314+0000","last_clean":"2024-05-06T00:30:40.725314+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:30:40.725314+0000","last_undegraded":"2024-05-06T00:30:40.725314+0000","last_fullsized":"2024-05-06T00:30:40.725314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:17.412771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012177069999999999,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":8762,"num_read_kb":50019,"num_write":296,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"199'456","reported_seq":4006,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:30.637866+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:30.558304+0000","last_became_peered":"2024-05-06T00:30:30.558304+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:58:28.298863+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.058990953999999998,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3578,"num_read_kb":12907,"num_write":404,"num_write_kb":3281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"199'918","reported_seq":9351,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667663+0000","last_change":"2024-05-06T00:30:39.656204+0000","last_active":"2024-05-06T00:30:40.667663+0000","last_peered":"2024-05-06T00:30:40.667663+0000","last_clean":"2024-05-06T00:30:40.667663+0000","last_became_active":"2024-05-06T00:30:39.653000+0000","last_became_peered":"2024-05-06T00:30:39.653000+0000","last_unstale":"2024-05-06T00:30:40.667663+0000","last_undegraded":"2024-05-06T00:30:40.667663+0000","last_fullsized":"2024-05-06T00:30:40.667663+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:11:01.551395+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.035914631000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":15,"num_read":12749,"num_read_kb":34289,"num_write":814,"num_write_kb":5275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"199'1204","reported_seq":10121,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725286+0000","last_change":"2024-05-06T00:30:30.704483+0000","last_active":"2024-05-06T00:30:40.725286+0000","last_peered":"2024-05-06T00:30:40.725286+0000","last_clean":"2024-05-06T00:30:40.725286+0000","last_became_active":"2024-05-06T00:30:30.568390+0000","last_became_peered":"2024-05-06T00:30:30.568390+0000","last_unstale":"2024-05-06T00:30:40.725286+0000","last_undegraded":"2024-05-06T00:30:40.725286+0000","last_fullsized":"2024-05-06T00:30:40.725286+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:14:14.323344+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.044642163999999998,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13696,"num_read_kb":33605,"num_write":1130,"num_write_kb":8108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"199'407","reported_seq":6998,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725360+0000","last_change":"2024-05-06T00:30:30.665158+0000","last_active":"2024-05-06T00:30:40.725360+0000","last_peered":"2024-05-06T00:30:40.725360+0000","last_clean":"2024-05-06T00:30:40.725360+0000","last_became_active":"2024-05-06T00:30:30.543930+0000","last_became_peered":"2024-05-06T00:30:30.543930+0000","last_unstale":"2024-05-06T00:30:40.725360+0000","last_undegraded":"2024-05-06T00:30:40.725360+0000","last_fullsized":"2024-05-06T00:30:40.725360+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:42:25.940056+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.040220385999999997,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":7280,"num_read_kb":25626,"num_write":414,"num_write_kb":13645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'681","reported_seq":3865,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668064+0000","last_change":"2024-05-06T00:30:30.639087+0000","last_active":"2024-05-06T00:30:40.668064+0000","last_peered":"2024-05-06T00:30:40.668064+0000","last_clean":"2024-05-06T00:30:40.668064+0000","last_became_active":"2024-05-06T00:30:30.560389+0000","last_became_peered":"2024-05-06T00:30:30.560389+0000","last_unstale":"2024-05-06T00:30:40.668064+0000","last_undegraded":"2024-05-06T00:30:40.668064+0000","last_fullsized":"2024-05-06T00:30:40.668064+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:47:13.586263+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.031279479999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5123,"num_read_kb":16860,"num_write":822,"num_write_kb":5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"185'748","reported_seq":14689,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725370+0000","last_change":"2024-05-06T00:30:35.598576+0000","last_active":"2024-05-06T00:30:40.725370+0000","last_peered":"2024-05-06T00:30:40.725370+0000","last_clean":"2024-05-06T00:30:40.725370+0000","last_became_active":"2024-05-06T00:30:35.595815+0000","last_became_peered":"2024-05-06T00:30:35.595815+0000","last_unstale":"2024-05-06T00:30:40.725370+0000","last_undegraded":"2024-05-06T00:30:40.725370+0000","last_fullsized":"2024-05-06T00:30:40.725370+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:45:37.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017909320000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":14404,"num_read_kb":28468,"num_write":531,"num_write_kb":3442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"194'349","reported_seq":4923,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725623+0000","last_change":"2024-05-06T00:30:19.957321+0000","last_active":"2024-05-06T00:30:40.725623+0000","last_peered":"2024-05-06T00:30:40.725623+0000","last_clean":"2024-05-06T00:30:40.725623+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:30:40.725623+0000","last_undegraded":"2024-05-06T00:30:40.725623+0000","last_fullsized":"2024-05-06T00:30:40.725623+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:27:18.137701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003171063,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3664,"num_read_kb":19738,"num_write":297,"num_write_kb":5164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"191'520","reported_seq":2900,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667838+0000","last_change":"2024-05-06T00:30:19.944378+0000","last_active":"2024-05-06T00:30:40.667838+0000","last_peered":"2024-05-06T00:30:40.667838+0000","last_clean":"2024-05-06T00:30:40.667838+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:30:40.667838+0000","last_undegraded":"2024-05-06T00:30:40.667838+0000","last_fullsized":"2024-05-06T00:30:40.667838+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:02:49.118813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004495626,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2722,"num_read_kb":15200,"num_write":616,"num_write_kb":1916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"191'298","reported_seq":5536,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665623+0000","last_change":"2024-05-06T00:30:36.646099+0000","last_active":"2024-05-06T00:30:40.665623+0000","last_peered":"2024-05-06T00:30:40.665623+0000","last_clean":"2024-05-06T00:30:40.665623+0000","last_became_active":"2024-05-06T00:30:36.641378+0000","last_became_peered":"2024-05-06T00:30:36.641378+0000","last_unstale":"2024-05-06T00:30:40.665623+0000","last_undegraded":"2024-05-06T00:30:40.665623+0000","last_fullsized":"2024-05-06T00:30:40.665623+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:03:14.811762+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031319316999999999,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":4880,"num_read_kb":11906,"num_write":264,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":76800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"199'529","reported_seq":7922,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665809+0000","last_change":"2024-05-06T00:30:36.655087+0000","last_active":"2024-05-06T00:30:40.665809+0000","last_peered":"2024-05-06T00:30:40.665809+0000","last_clean":"2024-05-06T00:30:40.665809+0000","last_became_active":"2024-05-06T00:30:36.651695+0000","last_became_peered":"2024-05-06T00:30:36.651695+0000","last_unstale":"2024-05-06T00:30:40.665809+0000","last_undegraded":"2024-05-06T00:30:40.665809+0000","last_fullsized":"2024-05-06T00:30:40.665809+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:30.279994+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.030926555000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":7962,"num_read_kb":31384,"num_write":673,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":237,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666855+0000","last_change":"2024-05-06T00:30:36.653124+0000","last_active":"2024-05-06T00:30:40.666855+0000","last_peered":"2024-05-06T00:30:40.666855+0000","last_clean":"2024-05-06T00:30:40.666855+0000","last_became_active":"2024-05-06T00:30:36.652331+0000","last_became_peered":"2024-05-06T00:30:36.652331+0000","last_unstale":"2024-05-06T00:30:40.666855+0000","last_undegraded":"2024-05-06T00:30:40.666855+0000","last_fullsized":"2024-05-06T00:30:40.666855+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:45:22.047908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"199'1005","reported_seq":7890,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725264+0000","last_change":"2024-05-06T00:30:30.709509+0000","last_active":"2024-05-06T00:30:40.725264+0000","last_peered":"2024-05-06T00:30:40.725264+0000","last_clean":"2024-05-06T00:30:40.725264+0000","last_became_active":"2024-05-06T00:30:30.569873+0000","last_became_peered":"2024-05-06T00:30:30.569873+0000","last_unstale":"2024-05-06T00:30:40.725264+0000","last_undegraded":"2024-05-06T00:30:40.725264+0000","last_fullsized":"2024-05-06T00:30:40.725264+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:37:31.390169+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.024654692999999998,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":10870,"num_read_kb":29913,"num_write":1138,"num_write_kb":7394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668489+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:30:40.668489+0000","last_peered":"2024-05-06T00:30:40.668489+0000","last_clean":"2024-05-06T00:30:40.668489+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:30:40.668489+0000","last_undegraded":"2024-05-06T00:30:40.668489+0000","last_fullsized":"2024-05-06T00:30:40.668489+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"191'766","reported_seq":6743,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725403+0000","last_change":"2024-05-06T00:30:35.593017+0000","last_active":"2024-05-06T00:30:40.725403+0000","last_peered":"2024-05-06T00:30:40.725403+0000","last_clean":"2024-05-06T00:30:40.725403+0000","last_became_active":"2024-05-06T00:30:35.589029+0000","last_became_peered":"2024-05-06T00:30:35.589029+0000","last_unstale":"2024-05-06T00:30:40.725403+0000","last_undegraded":"2024-05-06T00:30:40.725403+0000","last_fullsized":"2024-05-06T00:30:40.725403+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:18:10.559148+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.003902722,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8939,"num_read_kb":19825,"num_write":709,"num_write_kb":2787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666154+0000","last_change":"2024-05-06T00:30:36.646507+0000","last_active":"2024-05-06T00:30:40.666154+0000","last_peered":"2024-05-06T00:30:40.666154+0000","last_clean":"2024-05-06T00:30:40.666154+0000","last_became_active":"2024-05-06T00:30:36.644967+0000","last_became_peered":"2024-05-06T00:30:36.644967+0000","last_unstale":"2024-05-06T00:30:40.666154+0000","last_undegraded":"2024-05-06T00:30:40.666154+0000","last_fullsized":"2024-05-06T00:30:40.666154+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:16:47.890394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"191'657","reported_seq":9441,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668189+0000","last_change":"2024-05-06T00:30:19.945917+0000","last_active":"2024-05-06T00:30:40.668189+0000","last_peered":"2024-05-06T00:30:40.668189+0000","last_clean":"2024-05-06T00:30:40.668189+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:30:40.668189+0000","last_undegraded":"2024-05-06T00:30:40.668189+0000","last_fullsized":"2024-05-06T00:30:40.668189+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:49:12.946087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060800150000000002,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":10036,"num_read_kb":30901,"num_write":694,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668452+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:30:40.668452+0000","last_peered":"2024-05-06T00:30:40.668452+0000","last_clean":"2024-05-06T00:30:40.668452+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:30:40.668452+0000","last_undegraded":"2024-05-06T00:30:40.668452+0000","last_fullsized":"2024-05-06T00:30:40.668452+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"191'559","reported_seq":5683,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667793+0000","last_change":"2024-05-06T00:30:19.941179+0000","last_active":"2024-05-06T00:30:40.667793+0000","last_peered":"2024-05-06T00:30:40.667793+0000","last_clean":"2024-05-06T00:30:40.667793+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:30:40.667793+0000","last_undegraded":"2024-05-06T00:30:40.667793+0000","last_fullsized":"2024-05-06T00:30:40.667793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:38:10.377797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016637620000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":6637,"num_read_kb":16275,"num_write":486,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"199'538","reported_seq":4051,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725423+0000","last_change":"2024-05-06T00:30:30.742622+0000","last_active":"2024-05-06T00:30:40.725423+0000","last_peered":"2024-05-06T00:30:40.725423+0000","last_clean":"2024-05-06T00:30:40.725423+0000","last_became_active":"2024-05-06T00:30:30.561444+0000","last_became_peered":"2024-05-06T00:30:30.561444+0000","last_unstale":"2024-05-06T00:30:40.725423+0000","last_undegraded":"2024-05-06T00:30:40.725423+0000","last_fullsized":"2024-05-06T00:30:40.725423+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:39:30.619891+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.034165040000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":17,"num_read":3499,"num_read_kb":25511,"num_write":548,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":105,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667480+0000","last_change":"2024-05-06T00:30:39.648249+0000","last_active":"2024-05-06T00:30:40.667480+0000","last_peered":"2024-05-06T00:30:40.667480+0000","last_clean":"2024-05-06T00:30:40.667480+0000","last_became_active":"2024-05-06T00:30:39.647436+0000","last_became_peered":"2024-05-06T00:30:39.647436+0000","last_unstale":"2024-05-06T00:30:40.667480+0000","last_undegraded":"2024-05-06T00:30:40.667480+0000","last_fullsized":"2024-05-06T00:30:40.667480+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:15:38.102710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"192'657","reported_seq":6870,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667981+0000","last_change":"2024-05-06T00:30:19.944664+0000","last_active":"2024-05-06T00:30:40.667981+0000","last_peered":"2024-05-06T00:30:40.667981+0000","last_clean":"2024-05-06T00:30:40.667981+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:30:40.667981+0000","last_undegraded":"2024-05-06T00:30:40.667981+0000","last_fullsized":"2024-05-06T00:30:40.667981+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:56:10.880278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051286049999999996,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7148,"num_read_kb":27106,"num_write":776,"num_write_kb":4745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665403+0000","last_change":"2024-05-06T00:30:36.644291+0000","last_active":"2024-05-06T00:30:40.665403+0000","last_peered":"2024-05-06T00:30:40.665403+0000","last_clean":"2024-05-06T00:30:40.665403+0000","last_became_active":"2024-05-06T00:30:36.643456+0000","last_became_peered":"2024-05-06T00:30:36.643456+0000","last_unstale":"2024-05-06T00:30:40.665403+0000","last_undegraded":"2024-05-06T00:30:40.665403+0000","last_fullsized":"2024-05-06T00:30:40.665403+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:25:26.257376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"199'383","reported_seq":4994,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725003+0000","last_change":"2024-05-06T00:30:30.656302+0000","last_active":"2024-05-06T00:30:40.725003+0000","last_peered":"2024-05-06T00:30:40.725003+0000","last_clean":"2024-05-06T00:30:40.725003+0000","last_became_active":"2024-05-06T00:30:30.562288+0000","last_became_peered":"2024-05-06T00:30:30.562288+0000","last_unstale":"2024-05-06T00:30:40.725003+0000","last_undegraded":"2024-05-06T00:30:40.725003+0000","last_fullsized":"2024-05-06T00:30:40.725003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:53:00.473185+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.028356438000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":4034,"num_read_kb":14959,"num_write":372,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665364+0000","last_change":"2024-05-06T00:30:36.643966+0000","last_active":"2024-05-06T00:30:40.665364+0000","last_peered":"2024-05-06T00:30:40.665364+0000","last_clean":"2024-05-06T00:30:40.665364+0000","last_became_active":"2024-05-06T00:30:36.642148+0000","last_became_peered":"2024-05-06T00:30:36.642148+0000","last_unstale":"2024-05-06T00:30:40.665364+0000","last_undegraded":"2024-05-06T00:30:40.665364+0000","last_fullsized":"2024-05-06T00:30:40.665364+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:19:16.556193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"193'778","reported_seq":10467,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668186+0000","last_change":"2024-05-06T00:30:19.947559+0000","last_active":"2024-05-06T00:30:40.668186+0000","last_peered":"2024-05-06T00:30:40.668186+0000","last_clean":"2024-05-06T00:30:40.668186+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:30:40.668186+0000","last_undegraded":"2024-05-06T00:30:40.668186+0000","last_fullsized":"2024-05-06T00:30:40.668186+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:23:03.571190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065619830000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":12206,"num_read_kb":24770,"num_write":877,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667373+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:30:40.667373+0000","last_peered":"2024-05-06T00:30:40.667373+0000","last_clean":"2024-05-06T00:30:40.667373+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:30:40.667373+0000","last_undegraded":"2024-05-06T00:30:40.667373+0000","last_fullsized":"2024-05-06T00:30:40.667373+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":86,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665277+0000","last_change":"2024-05-06T00:30:36.642156+0000","last_active":"2024-05-06T00:30:40.665277+0000","last_peered":"2024-05-06T00:30:40.665277+0000","last_clean":"2024-05-06T00:30:40.665277+0000","last_became_active":"2024-05-06T00:30:36.641260+0000","last_became_peered":"2024-05-06T00:30:36.641260+0000","last_unstale":"2024-05-06T00:30:40.665277+0000","last_undegraded":"2024-05-06T00:30:40.665277+0000","last_fullsized":"2024-05-06T00:30:40.665277+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:31.624557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"191'620","reported_seq":6214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725308+0000","last_change":"2024-05-06T00:30:35.595715+0000","last_active":"2024-05-06T00:30:40.725308+0000","last_peered":"2024-05-06T00:30:40.725308+0000","last_clean":"2024-05-06T00:30:40.725308+0000","last_became_active":"2024-05-06T00:30:35.591860+0000","last_became_peered":"2024-05-06T00:30:35.591860+0000","last_unstale":"2024-05-06T00:30:40.725308+0000","last_undegraded":"2024-05-06T00:30:40.725308+0000","last_fullsized":"2024-05-06T00:30:40.725308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:47:33.295174+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.021797200999999999,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":6539,"num_read_kb":27944,"num_write":612,"num_write_kb":8155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"199'575","reported_seq":5828,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725392+0000","last_change":"2024-05-06T00:30:30.675754+0000","last_active":"2024-05-06T00:30:40.725392+0000","last_peered":"2024-05-06T00:30:40.725392+0000","last_clean":"2024-05-06T00:30:40.725392+0000","last_became_active":"2024-05-06T00:30:30.545607+0000","last_became_peered":"2024-05-06T00:30:30.545607+0000","last_unstale":"2024-05-06T00:30:40.725392+0000","last_undegraded":"2024-05-06T00:30:40.725392+0000","last_fullsized":"2024-05-06T00:30:40.725392+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:54:11.975131+0000","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.036169352000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":5497,"num_read_kb":27780,"num_write":687,"num_write_kb":13301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"199'805","reported_seq":8521,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665648+0000","last_change":"2024-05-06T00:30:36.656283+0000","last_active":"2024-05-06T00:30:40.665648+0000","last_peered":"2024-05-06T00:30:40.665648+0000","last_clean":"2024-05-06T00:30:40.665648+0000","last_became_active":"2024-05-06T00:30:36.651797+0000","last_became_peered":"2024-05-06T00:30:36.651797+0000","last_unstale":"2024-05-06T00:30:40.665648+0000","last_undegraded":"2024-05-06T00:30:40.665648+0000","last_fullsized":"2024-05-06T00:30:40.665648+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:47:48.879526+0000","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0.028632154999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":25,"num_read":9542,"num_read_kb":44861,"num_write":766,"num_write_kb":8347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"187'351","reported_seq":7154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668166+0000","last_change":"2024-05-06T00:30:19.946118+0000","last_active":"2024-05-06T00:30:40.668166+0000","last_peered":"2024-05-06T00:30:40.668166+0000","last_clean":"2024-05-06T00:30:40.668166+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:30:40.668166+0000","last_undegraded":"2024-05-06T00:30:40.668166+0000","last_fullsized":"2024-05-06T00:30:40.668166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:06:25.304781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057155160000000003,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5046,"num_read_kb":27757,"num_write":349,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"188'420","reported_seq":6843,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:39.652878+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:39.650059+0000","last_became_peered":"2024-05-06T00:30:39.650059+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:14:23.610386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057267869999999997,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":6116,"num_read_kb":54371,"num_write":370,"num_write_kb":8906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"187'890","reported_seq":12713,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666252+0000","last_change":"2024-05-06T00:30:36.650116+0000","last_active":"2024-05-06T00:30:40.666252+0000","last_peered":"2024-05-06T00:30:40.666252+0000","last_clean":"2024-05-06T00:30:40.666252+0000","last_became_active":"2024-05-06T00:30:36.645606+0000","last_became_peered":"2024-05-06T00:30:36.645606+0000","last_unstale":"2024-05-06T00:30:40.666252+0000","last_undegraded":"2024-05-06T00:30:40.666252+0000","last_fullsized":"2024-05-06T00:30:40.666252+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:34:38.495321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015459429999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":13504,"num_read_kb":33376,"num_write":1160,"num_write_kb":4653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"199'867","reported_seq":12481,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668014+0000","last_change":"2024-05-06T00:30:39.656310+0000","last_active":"2024-05-06T00:30:40.668014+0000","last_peered":"2024-05-06T00:30:40.668014+0000","last_clean":"2024-05-06T00:30:40.668014+0000","last_became_active":"2024-05-06T00:30:39.652991+0000","last_became_peered":"2024-05-06T00:30:39.652991+0000","last_unstale":"2024-05-06T00:30:40.668014+0000","last_undegraded":"2024-05-06T00:30:40.668014+0000","last_fullsized":"2024-05-06T00:30:40.668014+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:35:14.638153+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.035058351000000001,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":15156,"num_read_kb":31725,"num_write":1055,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"190'348","reported_seq":4259,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668185+0000","last_change":"2024-05-06T00:30:19.943335+0000","last_active":"2024-05-06T00:30:40.668185+0000","last_peered":"2024-05-06T00:30:40.668185+0000","last_clean":"2024-05-06T00:30:40.668185+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:30:40.668185+0000","last_undegraded":"2024-05-06T00:30:40.668185+0000","last_fullsized":"2024-05-06T00:30:40.668185+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:42:14.046101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033172480000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":3496,"num_read_kb":19343,"num_write":377,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'790","reported_seq":11712,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667348+0000","last_change":"2024-05-06T00:30:39.652246+0000","last_active":"2024-05-06T00:30:40.667348+0000","last_peered":"2024-05-06T00:30:40.667348+0000","last_clean":"2024-05-06T00:30:40.667348+0000","last_became_active":"2024-05-06T00:30:39.649412+0000","last_became_peered":"2024-05-06T00:30:39.649412+0000","last_unstale":"2024-05-06T00:30:40.667348+0000","last_undegraded":"2024-05-06T00:30:40.667348+0000","last_fullsized":"2024-05-06T00:30:40.667348+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:51:30.132753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001791927,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13318,"num_read_kb":32027,"num_write":733,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"199'437","reported_seq":5748,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665489+0000","last_change":"2024-05-06T00:30:36.655967+0000","last_active":"2024-05-06T00:30:40.665489+0000","last_peered":"2024-05-06T00:30:40.665489+0000","last_clean":"2024-05-06T00:30:40.665489+0000","last_became_active":"2024-05-06T00:30:36.652383+0000","last_became_peered":"2024-05-06T00:30:36.652383+0000","last_unstale":"2024-05-06T00:30:40.665489+0000","last_undegraded":"2024-05-06T00:30:40.665489+0000","last_fullsized":"2024-05-06T00:30:40.665489+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:56:34.123681+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.029993888999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5408,"num_read_kb":10751,"num_write":542,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"191'477","reported_seq":9126,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725278+0000","last_change":"2024-05-06T00:30:19.956297+0000","last_active":"2024-05-06T00:30:40.725278+0000","last_peered":"2024-05-06T00:30:40.725278+0000","last_clean":"2024-05-06T00:30:40.725278+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:30:40.725278+0000","last_undegraded":"2024-05-06T00:30:40.725278+0000","last_fullsized":"2024-05-06T00:30:40.725278+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:46.269848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002988648,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8695,"num_read_kb":23092,"num_write":389,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"192'673","reported_seq":9486,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665601+0000","last_change":"2024-05-06T00:30:36.651926+0000","last_active":"2024-05-06T00:30:40.665601+0000","last_peered":"2024-05-06T00:30:40.665601+0000","last_clean":"2024-05-06T00:30:40.665601+0000","last_became_active":"2024-05-06T00:30:36.647614+0000","last_became_peered":"2024-05-06T00:30:36.647614+0000","last_unstale":"2024-05-06T00:30:40.665601+0000","last_undegraded":"2024-05-06T00:30:40.665601+0000","last_fullsized":"2024-05-06T00:30:40.665601+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:16:00.581426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00351067,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":10781,"num_read_kb":40700,"num_write":619,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":623,"num_object_clones":0,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":619,"num_read":263747,"num_read_kb":883054,"num_write":20048,"num_write_kb":192626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5734,"num_bytes_recovered":72015234,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},"log_size":19745,"ondisk_log_size":19745,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640313,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60032,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311808,"statfs":{"total":96636764160,"available":96575291392,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":198,"seq":850403524613,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":2476,"kb_used_omap":44,"kb_used_meta":29331,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45664,"internal_metadata":30035360},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640311,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640312,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47544,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324296,"statfs":{"total":96636764160,"available":96588079104,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640313,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47552,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324288,"statfs":{"total":96636764160,"available":96588070912,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640311,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48132,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323708,"statfs":{"total":96636764160,"available":96587476992,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:30:44.729 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:30:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:45.015 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:30:45.031 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-05-06T00:30:44.461212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":625,"num_object_clones":0,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":619,"num_read":263813,"num_read_kb":883109,"num_write":20149,"num_write_kb":194254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5740,"num_bytes_recovered":73786338,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19817,"ondisk_log_size":19817,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":283248,"kb_used_data":16000,"kb_used_omap":52,"kb_used_meta":267083,"kb_avail":565947792,"statfs":{"total":579820584960,"available":579530539008,"internally_reserved":0,"allocated":16384000,"data_stored":7584694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53637,"internal_metadata":273493627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001783"},"pg_stats":[{"pgid":"3.1f","version":"194'672","reported_seq":13023,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725510+0000","last_change":"2024-05-06T00:30:19.956846+0000","last_active":"2024-05-06T00:30:40.725510+0000","last_peered":"2024-05-06T00:30:40.725510+0000","last_clean":"2024-05-06T00:30:40.725510+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:30:40.725510+0000","last_undegraded":"2024-05-06T00:30:40.725510+0000","last_fullsized":"2024-05-06T00:30:40.725510+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:12:04.053936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028964080000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":12460,"num_read_kb":40065,"num_write":528,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"191'367","reported_seq":9956,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725314+0000","last_change":"2024-05-06T00:30:19.973477+0000","last_active":"2024-05-06T00:30:40.725314+0000","last_peered":"2024-05-06T00:30:40.725314+0000","last_clean":"2024-05-06T00:30:40.725314+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:30:40.725314+0000","last_undegraded":"2024-05-06T00:30:40.725314+0000","last_fullsized":"2024-05-06T00:30:40.725314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:17.412771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012177069999999999,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":8762,"num_read_kb":50019,"num_write":296,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"199'456","reported_seq":4006,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:30.637866+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:30.558304+0000","last_became_peered":"2024-05-06T00:30:30.558304+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:58:28.298863+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.058990953999999998,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3578,"num_read_kb":12907,"num_write":404,"num_write_kb":3281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"199'918","reported_seq":9351,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667663+0000","last_change":"2024-05-06T00:30:39.656204+0000","last_active":"2024-05-06T00:30:40.667663+0000","last_peered":"2024-05-06T00:30:40.667663+0000","last_clean":"2024-05-06T00:30:40.667663+0000","last_became_active":"2024-05-06T00:30:39.653000+0000","last_became_peered":"2024-05-06T00:30:39.653000+0000","last_unstale":"2024-05-06T00:30:40.667663+0000","last_undegraded":"2024-05-06T00:30:40.667663+0000","last_fullsized":"2024-05-06T00:30:40.667663+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:11:01.551395+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.035914631000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":15,"num_read":12749,"num_read_kb":34289,"num_write":814,"num_write_kb":5275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"199'1204","reported_seq":10121,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725286+0000","last_change":"2024-05-06T00:30:30.704483+0000","last_active":"2024-05-06T00:30:40.725286+0000","last_peered":"2024-05-06T00:30:40.725286+0000","last_clean":"2024-05-06T00:30:40.725286+0000","last_became_active":"2024-05-06T00:30:30.568390+0000","last_became_peered":"2024-05-06T00:30:30.568390+0000","last_unstale":"2024-05-06T00:30:40.725286+0000","last_undegraded":"2024-05-06T00:30:40.725286+0000","last_fullsized":"2024-05-06T00:30:40.725286+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:14:14.323344+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.044642163999999998,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13696,"num_read_kb":33605,"num_write":1130,"num_write_kb":8108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"199'407","reported_seq":6998,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725360+0000","last_change":"2024-05-06T00:30:30.665158+0000","last_active":"2024-05-06T00:30:40.725360+0000","last_peered":"2024-05-06T00:30:40.725360+0000","last_clean":"2024-05-06T00:30:40.725360+0000","last_became_active":"2024-05-06T00:30:30.543930+0000","last_became_peered":"2024-05-06T00:30:30.543930+0000","last_unstale":"2024-05-06T00:30:40.725360+0000","last_undegraded":"2024-05-06T00:30:40.725360+0000","last_fullsized":"2024-05-06T00:30:40.725360+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:42:25.940056+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.040220385999999997,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":7280,"num_read_kb":25626,"num_write":414,"num_write_kb":13645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'681","reported_seq":3865,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668064+0000","last_change":"2024-05-06T00:30:30.639087+0000","last_active":"2024-05-06T00:30:40.668064+0000","last_peered":"2024-05-06T00:30:40.668064+0000","last_clean":"2024-05-06T00:30:40.668064+0000","last_became_active":"2024-05-06T00:30:30.560389+0000","last_became_peered":"2024-05-06T00:30:30.560389+0000","last_unstale":"2024-05-06T00:30:40.668064+0000","last_undegraded":"2024-05-06T00:30:40.668064+0000","last_fullsized":"2024-05-06T00:30:40.668064+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:47:13.586263+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.031279479999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5123,"num_read_kb":16860,"num_write":822,"num_write_kb":5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"185'748","reported_seq":14689,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725370+0000","last_change":"2024-05-06T00:30:35.598576+0000","last_active":"2024-05-06T00:30:40.725370+0000","last_peered":"2024-05-06T00:30:40.725370+0000","last_clean":"2024-05-06T00:30:40.725370+0000","last_became_active":"2024-05-06T00:30:35.595815+0000","last_became_peered":"2024-05-06T00:30:35.595815+0000","last_unstale":"2024-05-06T00:30:40.725370+0000","last_undegraded":"2024-05-06T00:30:40.725370+0000","last_fullsized":"2024-05-06T00:30:40.725370+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:45:37.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017909320000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":14404,"num_read_kb":28468,"num_write":531,"num_write_kb":3442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"194'349","reported_seq":4923,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725623+0000","last_change":"2024-05-06T00:30:19.957321+0000","last_active":"2024-05-06T00:30:40.725623+0000","last_peered":"2024-05-06T00:30:40.725623+0000","last_clean":"2024-05-06T00:30:40.725623+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:30:40.725623+0000","last_undegraded":"2024-05-06T00:30:40.725623+0000","last_fullsized":"2024-05-06T00:30:40.725623+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:27:18.137701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003171063,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3664,"num_read_kb":19738,"num_write":297,"num_write_kb":5164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"191'520","reported_seq":2900,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667838+0000","last_change":"2024-05-06T00:30:19.944378+0000","last_active":"2024-05-06T00:30:40.667838+0000","last_peered":"2024-05-06T00:30:40.667838+0000","last_clean":"2024-05-06T00:30:40.667838+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:30:40.667838+0000","last_undegraded":"2024-05-06T00:30:40.667838+0000","last_fullsized":"2024-05-06T00:30:40.667838+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:02:49.118813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004495626,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2722,"num_read_kb":15200,"num_write":616,"num_write_kb":1916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"191'298","reported_seq":5536,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665623+0000","last_change":"2024-05-06T00:30:36.646099+0000","last_active":"2024-05-06T00:30:40.665623+0000","last_peered":"2024-05-06T00:30:40.665623+0000","last_clean":"2024-05-06T00:30:40.665623+0000","last_became_active":"2024-05-06T00:30:36.641378+0000","last_became_peered":"2024-05-06T00:30:36.641378+0000","last_unstale":"2024-05-06T00:30:40.665623+0000","last_undegraded":"2024-05-06T00:30:40.665623+0000","last_fullsized":"2024-05-06T00:30:40.665623+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:03:14.811762+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031319316999999999,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":4880,"num_read_kb":11906,"num_write":264,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":76800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"199'529","reported_seq":7922,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665809+0000","last_change":"2024-05-06T00:30:36.655087+0000","last_active":"2024-05-06T00:30:40.665809+0000","last_peered":"2024-05-06T00:30:40.665809+0000","last_clean":"2024-05-06T00:30:40.665809+0000","last_became_active":"2024-05-06T00:30:36.651695+0000","last_became_peered":"2024-05-06T00:30:36.651695+0000","last_unstale":"2024-05-06T00:30:40.665809+0000","last_undegraded":"2024-05-06T00:30:40.665809+0000","last_fullsized":"2024-05-06T00:30:40.665809+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:30.279994+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.030926555000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":7962,"num_read_kb":31384,"num_write":673,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":237,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666855+0000","last_change":"2024-05-06T00:30:36.653124+0000","last_active":"2024-05-06T00:30:40.666855+0000","last_peered":"2024-05-06T00:30:40.666855+0000","last_clean":"2024-05-06T00:30:40.666855+0000","last_became_active":"2024-05-06T00:30:36.652331+0000","last_became_peered":"2024-05-06T00:30:36.652331+0000","last_unstale":"2024-05-06T00:30:40.666855+0000","last_undegraded":"2024-05-06T00:30:40.666855+0000","last_fullsized":"2024-05-06T00:30:40.666855+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:45:22.047908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"199'1005","reported_seq":7890,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725264+0000","last_change":"2024-05-06T00:30:30.709509+0000","last_active":"2024-05-06T00:30:40.725264+0000","last_peered":"2024-05-06T00:30:40.725264+0000","last_clean":"2024-05-06T00:30:40.725264+0000","last_became_active":"2024-05-06T00:30:30.569873+0000","last_became_peered":"2024-05-06T00:30:30.569873+0000","last_unstale":"2024-05-06T00:30:40.725264+0000","last_undegraded":"2024-05-06T00:30:40.725264+0000","last_fullsized":"2024-05-06T00:30:40.725264+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:37:31.390169+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.024654692999999998,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":10870,"num_read_kb":29913,"num_write":1138,"num_write_kb":7394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668489+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:30:40.668489+0000","last_peered":"2024-05-06T00:30:40.668489+0000","last_clean":"2024-05-06T00:30:40.668489+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:30:40.668489+0000","last_undegraded":"2024-05-06T00:30:40.668489+0000","last_fullsized":"2024-05-06T00:30:40.668489+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"191'766","reported_seq":6743,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725403+0000","last_change":"2024-05-06T00:30:35.593017+0000","last_active":"2024-05-06T00:30:40.725403+0000","last_peered":"2024-05-06T00:30:40.725403+0000","last_clean":"2024-05-06T00:30:40.725403+0000","last_became_active":"2024-05-06T00:30:35.589029+0000","last_became_peered":"2024-05-06T00:30:35.589029+0000","last_unstale":"2024-05-06T00:30:40.725403+0000","last_undegraded":"2024-05-06T00:30:40.725403+0000","last_fullsized":"2024-05-06T00:30:40.725403+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:18:10.559148+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.003902722,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8939,"num_read_kb":19825,"num_write":709,"num_write_kb":2787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666154+0000","last_change":"2024-05-06T00:30:36.646507+0000","last_active":"2024-05-06T00:30:40.666154+0000","last_peered":"2024-05-06T00:30:40.666154+0000","last_clean":"2024-05-06T00:30:40.666154+0000","last_became_active":"2024-05-06T00:30:36.644967+0000","last_became_peered":"2024-05-06T00:30:36.644967+0000","last_unstale":"2024-05-06T00:30:40.666154+0000","last_undegraded":"2024-05-06T00:30:40.666154+0000","last_fullsized":"2024-05-06T00:30:40.666154+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:16:47.890394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"191'657","reported_seq":9441,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668189+0000","last_change":"2024-05-06T00:30:19.945917+0000","last_active":"2024-05-06T00:30:40.668189+0000","last_peered":"2024-05-06T00:30:40.668189+0000","last_clean":"2024-05-06T00:30:40.668189+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:30:40.668189+0000","last_undegraded":"2024-05-06T00:30:40.668189+0000","last_fullsized":"2024-05-06T00:30:40.668189+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:49:12.946087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060800150000000002,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":10036,"num_read_kb":30901,"num_write":694,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668452+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:30:40.668452+0000","last_peered":"2024-05-06T00:30:40.668452+0000","last_clean":"2024-05-06T00:30:40.668452+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:30:40.668452+0000","last_undegraded":"2024-05-06T00:30:40.668452+0000","last_fullsized":"2024-05-06T00:30:40.668452+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"191'559","reported_seq":5683,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667793+0000","last_change":"2024-05-06T00:30:19.941179+0000","last_active":"2024-05-06T00:30:40.667793+0000","last_peered":"2024-05-06T00:30:40.667793+0000","last_clean":"2024-05-06T00:30:40.667793+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:30:40.667793+0000","last_undegraded":"2024-05-06T00:30:40.667793+0000","last_fullsized":"2024-05-06T00:30:40.667793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:38:10.377797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016637620000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":6637,"num_read_kb":16275,"num_write":486,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"199'538","reported_seq":4051,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725423+0000","last_change":"2024-05-06T00:30:30.742622+0000","last_active":"2024-05-06T00:30:40.725423+0000","last_peered":"2024-05-06T00:30:40.725423+0000","last_clean":"2024-05-06T00:30:40.725423+0000","last_became_active":"2024-05-06T00:30:30.561444+0000","last_became_peered":"2024-05-06T00:30:30.561444+0000","last_unstale":"2024-05-06T00:30:40.725423+0000","last_undegraded":"2024-05-06T00:30:40.725423+0000","last_fullsized":"2024-05-06T00:30:40.725423+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:39:30.619891+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.034165040000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":17,"num_read":3499,"num_read_kb":25511,"num_write":548,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":105,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667480+0000","last_change":"2024-05-06T00:30:39.648249+0000","last_active":"2024-05-06T00:30:40.667480+0000","last_peered":"2024-05-06T00:30:40.667480+0000","last_clean":"2024-05-06T00:30:40.667480+0000","last_became_active":"2024-05-06T00:30:39.647436+0000","last_became_peered":"2024-05-06T00:30:39.647436+0000","last_unstale":"2024-05-06T00:30:40.667480+0000","last_undegraded":"2024-05-06T00:30:40.667480+0000","last_fullsized":"2024-05-06T00:30:40.667480+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:15:38.102710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"192'657","reported_seq":6870,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667981+0000","last_change":"2024-05-06T00:30:19.944664+0000","last_active":"2024-05-06T00:30:40.667981+0000","last_peered":"2024-05-06T00:30:40.667981+0000","last_clean":"2024-05-06T00:30:40.667981+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:30:40.667981+0000","last_undegraded":"2024-05-06T00:30:40.667981+0000","last_fullsized":"2024-05-06T00:30:40.667981+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:56:10.880278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051286049999999996,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7148,"num_read_kb":27106,"num_write":776,"num_write_kb":4745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665403+0000","last_change":"2024-05-06T00:30:36.644291+0000","last_active":"2024-05-06T00:30:40.665403+0000","last_peered":"2024-05-06T00:30:40.665403+0000","last_clean":"2024-05-06T00:30:40.665403+0000","last_became_active":"2024-05-06T00:30:36.643456+0000","last_became_peered":"2024-05-06T00:30:36.643456+0000","last_unstale":"2024-05-06T00:30:40.665403+0000","last_undegraded":"2024-05-06T00:30:40.665403+0000","last_fullsized":"2024-05-06T00:30:40.665403+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:25:26.257376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"199'383","reported_seq":4994,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725003+0000","last_change":"2024-05-06T00:30:30.656302+0000","last_active":"2024-05-06T00:30:40.725003+0000","last_peered":"2024-05-06T00:30:40.725003+0000","last_clean":"2024-05-06T00:30:40.725003+0000","last_became_active":"2024-05-06T00:30:30.562288+0000","last_became_peered":"2024-05-06T00:30:30.562288+0000","last_unstale":"2024-05-06T00:30:40.725003+0000","last_undegraded":"2024-05-06T00:30:40.725003+0000","last_fullsized":"2024-05-06T00:30:40.725003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:53:00.473185+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.028356438000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":4034,"num_read_kb":14959,"num_write":372,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665364+0000","last_change":"2024-05-06T00:30:36.643966+0000","last_active":"2024-05-06T00:30:40.665364+0000","last_peered":"2024-05-06T00:30:40.665364+0000","last_clean":"2024-05-06T00:30:40.665364+0000","last_became_active":"2024-05-06T00:30:36.642148+0000","last_became_peered":"2024-05-06T00:30:36.642148+0000","last_unstale":"2024-05-06T00:30:40.665364+0000","last_undegraded":"2024-05-06T00:30:40.665364+0000","last_fullsized":"2024-05-06T00:30:40.665364+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:19:16.556193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"193'778","reported_seq":10467,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668186+0000","last_change":"2024-05-06T00:30:19.947559+0000","last_active":"2024-05-06T00:30:40.668186+0000","last_peered":"2024-05-06T00:30:40.668186+0000","last_clean":"2024-05-06T00:30:40.668186+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:30:40.668186+0000","last_undegraded":"2024-05-06T00:30:40.668186+0000","last_fullsized":"2024-05-06T00:30:40.668186+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:23:03.571190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065619830000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":12206,"num_read_kb":24770,"num_write":877,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667373+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:30:40.667373+0000","last_peered":"2024-05-06T00:30:40.667373+0000","last_clean":"2024-05-06T00:30:40.667373+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:30:40.667373+0000","last_undegraded":"2024-05-06T00:30:40.667373+0000","last_fullsized":"2024-05-06T00:30:40.667373+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":86,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665277+0000","last_change":"2024-05-06T00:30:36.642156+0000","last_active":"2024-05-06T00:30:40.665277+0000","last_peered":"2024-05-06T00:30:40.665277+0000","last_clean":"2024-05-06T00:30:40.665277+0000","last_became_active":"2024-05-06T00:30:36.641260+0000","last_became_peered":"2024-05-06T00:30:36.641260+0000","last_unstale":"2024-05-06T00:30:40.665277+0000","last_undegraded":"2024-05-06T00:30:40.665277+0000","last_fullsized":"2024-05-06T00:30:40.665277+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:31.624557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"191'620","reported_seq":6214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725308+0000","last_change":"2024-05-06T00:30:35.595715+0000","last_active":"2024-05-06T00:30:40.725308+0000","last_peered":"2024-05-06T00:30:40.725308+0000","last_clean":"2024-05-06T00:30:40.725308+0000","last_became_active":"2024-05-06T00:30:35.591860+0000","last_became_peered":"2024-05-06T00:30:35.591860+0000","last_unstale":"2024-05-06T00:30:40.725308+0000","last_undegraded":"2024-05-06T00:30:40.725308+0000","last_fullsized":"2024-05-06T00:30:40.725308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:47:33.295174+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.021797200999999999,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":6539,"num_read_kb":27944,"num_write":612,"num_write_kb":8155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"199'575","reported_seq":5828,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725392+0000","last_change":"2024-05-06T00:30:30.675754+0000","last_active":"2024-05-06T00:30:40.725392+0000","last_peered":"2024-05-06T00:30:40.725392+0000","last_clean":"2024-05-06T00:30:40.725392+0000","last_became_active":"2024-05-06T00:30:30.545607+0000","last_became_peered":"2024-05-06T00:30:30.545607+0000","last_unstale":"2024-05-06T00:30:40.725392+0000","last_undegraded":"2024-05-06T00:30:40.725392+0000","last_fullsized":"2024-05-06T00:30:40.725392+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:54:11.975131+0000","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.036169352000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":5497,"num_read_kb":27780,"num_write":687,"num_write_kb":13301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"199'805","reported_seq":8521,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665648+0000","last_change":"2024-05-06T00:30:36.656283+0000","last_active":"2024-05-06T00:30:40.665648+0000","last_peered":"2024-05-06T00:30:40.665648+0000","last_clean":"2024-05-06T00:30:40.665648+0000","last_became_active":"2024-05-06T00:30:36.651797+0000","last_became_peered":"2024-05-06T00:30:36.651797+0000","last_unstale":"2024-05-06T00:30:40.665648+0000","last_undegraded":"2024-05-06T00:30:40.665648+0000","last_fullsized":"2024-05-06T00:30:40.665648+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:47:48.879526+0000","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0.028632154999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":25,"num_read":9542,"num_read_kb":44861,"num_write":766,"num_write_kb":8347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"187'351","reported_seq":7154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668166+0000","last_change":"2024-05-06T00:30:19.946118+0000","last_active":"2024-05-06T00:30:40.668166+0000","last_peered":"2024-05-06T00:30:40.668166+0000","last_clean":"2024-05-06T00:30:40.668166+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:30:40.668166+0000","last_undegraded":"2024-05-06T00:30:40.668166+0000","last_fullsized":"2024-05-06T00:30:40.668166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:06:25.304781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057155160000000003,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5046,"num_read_kb":27757,"num_write":349,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"188'420","reported_seq":6843,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:39.652878+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:39.650059+0000","last_became_peered":"2024-05-06T00:30:39.650059+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:14:23.610386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057267869999999997,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":6116,"num_read_kb":54371,"num_write":370,"num_write_kb":8906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"187'890","reported_seq":12713,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666252+0000","last_change":"2024-05-06T00:30:36.650116+0000","last_active":"2024-05-06T00:30:40.666252+0000","last_peered":"2024-05-06T00:30:40.666252+0000","last_clean":"2024-05-06T00:30:40.666252+0000","last_became_active":"2024-05-06T00:30:36.645606+0000","last_became_peered":"2024-05-06T00:30:36.645606+0000","last_unstale":"2024-05-06T00:30:40.666252+0000","last_undegraded":"2024-05-06T00:30:40.666252+0000","last_fullsized":"2024-05-06T00:30:40.666252+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:34:38.495321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015459429999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":13504,"num_read_kb":33376,"num_write":1160,"num_write_kb":4653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"199'867","reported_seq":12481,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668014+0000","last_change":"2024-05-06T00:30:39.656310+0000","last_active":"2024-05-06T00:30:40.668014+0000","last_peered":"2024-05-06T00:30:40.668014+0000","last_clean":"2024-05-06T00:30:40.668014+0000","last_became_active":"2024-05-06T00:30:39.652991+0000","last_became_peered":"2024-05-06T00:30:39.652991+0000","last_unstale":"2024-05-06T00:30:40.668014+0000","last_undegraded":"2024-05-06T00:30:40.668014+0000","last_fullsized":"2024-05-06T00:30:40.668014+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:35:14.638153+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.035058351000000001,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":15156,"num_read_kb":31725,"num_write":1055,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"190'348","reported_seq":4259,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668185+0000","last_change":"2024-05-06T00:30:19.943335+0000","last_active":"2024-05-06T00:30:40.668185+0000","last_peered":"2024-05-06T00:30:40.668185+0000","last_clean":"2024-05-06T00:30:40.668185+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:30:40.668185+0000","last_undegraded":"2024-05-06T00:30:40.668185+0000","last_fullsized":"2024-05-06T00:30:40.668185+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:42:14.046101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033172480000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":3496,"num_read_kb":19343,"num_write":377,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'790","reported_seq":11712,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667348+0000","last_change":"2024-05-06T00:30:39.652246+0000","last_active":"2024-05-06T00:30:40.667348+0000","last_peered":"2024-05-06T00:30:40.667348+0000","last_clean":"2024-05-06T00:30:40.667348+0000","last_became_active":"2024-05-06T00:30:39.649412+0000","last_became_peered":"2024-05-06T00:30:39.649412+0000","last_unstale":"2024-05-06T00:30:40.667348+0000","last_undegraded":"2024-05-06T00:30:40.667348+0000","last_fullsized":"2024-05-06T00:30:40.667348+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:51:30.132753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001791927,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13318,"num_read_kb":32027,"num_write":733,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"199'437","reported_seq":5748,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665489+0000","last_change":"2024-05-06T00:30:36.655967+0000","last_active":"2024-05-06T00:30:40.665489+0000","last_peered":"2024-05-06T00:30:40.665489+0000","last_clean":"2024-05-06T00:30:40.665489+0000","last_became_active":"2024-05-06T00:30:36.652383+0000","last_became_peered":"2024-05-06T00:30:36.652383+0000","last_unstale":"2024-05-06T00:30:40.665489+0000","last_undegraded":"2024-05-06T00:30:40.665489+0000","last_fullsized":"2024-05-06T00:30:40.665489+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:56:34.123681+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.029993888999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5408,"num_read_kb":10751,"num_write":542,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"191'477","reported_seq":9126,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725278+0000","last_change":"2024-05-06T00:30:19.956297+0000","last_active":"2024-05-06T00:30:40.725278+0000","last_peered":"2024-05-06T00:30:40.725278+0000","last_clean":"2024-05-06T00:30:40.725278+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:30:40.725278+0000","last_undegraded":"2024-05-06T00:30:40.725278+0000","last_fullsized":"2024-05-06T00:30:40.725278+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:46.269848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002988648,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8695,"num_read_kb":23092,"num_write":389,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"192'673","reported_seq":9486,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665601+0000","last_change":"2024-05-06T00:30:36.651926+0000","last_active":"2024-05-06T00:30:40.665601+0000","last_peered":"2024-05-06T00:30:40.665601+0000","last_clean":"2024-05-06T00:30:40.665601+0000","last_became_active":"2024-05-06T00:30:36.647614+0000","last_became_peered":"2024-05-06T00:30:36.647614+0000","last_unstale":"2024-05-06T00:30:40.665601+0000","last_undegraded":"2024-05-06T00:30:40.665601+0000","last_fullsized":"2024-05-06T00:30:40.665601+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:16:00.581426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00351067,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":10781,"num_read_kb":40700,"num_write":619,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":623,"num_object_clones":0,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":619,"num_read":263747,"num_read_kb":883054,"num_write":20048,"num_write_kb":192626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5734,"num_bytes_recovered":72015234,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},"log_size":19745,"ondisk_log_size":19745,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640313,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60032,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311808,"statfs":{"total":96636764160,"available":96575291392,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":198,"seq":850403524613,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":2476,"kb_used_omap":44,"kb_used_meta":29331,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45664,"internal_metadata":30035360},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640311,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640312,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47544,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324296,"statfs":{"total":96636764160,"available":96588079104,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640313,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47552,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324288,"statfs":{"total":96636764160,"available":96588070912,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640311,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48132,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323708,"statfs":{"total":96636764160,"available":96587476992,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:30:45.032 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-06T00:30:45.033 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:30:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:45.309 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:30:45.325 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-05-06T00:30:44.461212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":625,"num_object_clones":0,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":619,"num_read":263813,"num_read_kb":883109,"num_write":20149,"num_write_kb":194254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5740,"num_bytes_recovered":73786338,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19817,"ondisk_log_size":19817,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":283248,"kb_used_data":16000,"kb_used_omap":52,"kb_used_meta":267083,"kb_avail":565947792,"statfs":{"total":579820584960,"available":579530539008,"internally_reserved":0,"allocated":16384000,"data_stored":7584694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53637,"internal_metadata":273493627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001783"},"pg_stats":[{"pgid":"3.1f","version":"194'672","reported_seq":13023,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725510+0000","last_change":"2024-05-06T00:30:19.956846+0000","last_active":"2024-05-06T00:30:40.725510+0000","last_peered":"2024-05-06T00:30:40.725510+0000","last_clean":"2024-05-06T00:30:40.725510+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:30:40.725510+0000","last_undegraded":"2024-05-06T00:30:40.725510+0000","last_fullsized":"2024-05-06T00:30:40.725510+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:12:04.053936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028964080000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":12460,"num_read_kb":40065,"num_write":528,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"191'367","reported_seq":9956,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725314+0000","last_change":"2024-05-06T00:30:19.973477+0000","last_active":"2024-05-06T00:30:40.725314+0000","last_peered":"2024-05-06T00:30:40.725314+0000","last_clean":"2024-05-06T00:30:40.725314+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:30:40.725314+0000","last_undegraded":"2024-05-06T00:30:40.725314+0000","last_fullsized":"2024-05-06T00:30:40.725314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:17.412771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012177069999999999,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":8762,"num_read_kb":50019,"num_write":296,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"199'456","reported_seq":4006,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:30.637866+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:30.558304+0000","last_became_peered":"2024-05-06T00:30:30.558304+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:58:28.298863+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.058990953999999998,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3578,"num_read_kb":12907,"num_write":404,"num_write_kb":3281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"199'918","reported_seq":9351,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667663+0000","last_change":"2024-05-06T00:30:39.656204+0000","last_active":"2024-05-06T00:30:40.667663+0000","last_peered":"2024-05-06T00:30:40.667663+0000","last_clean":"2024-05-06T00:30:40.667663+0000","last_became_active":"2024-05-06T00:30:39.653000+0000","last_became_peered":"2024-05-06T00:30:39.653000+0000","last_unstale":"2024-05-06T00:30:40.667663+0000","last_undegraded":"2024-05-06T00:30:40.667663+0000","last_fullsized":"2024-05-06T00:30:40.667663+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:11:01.551395+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.035914631000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":15,"num_read":12749,"num_read_kb":34289,"num_write":814,"num_write_kb":5275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"199'1204","reported_seq":10121,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725286+0000","last_change":"2024-05-06T00:30:30.704483+0000","last_active":"2024-05-06T00:30:40.725286+0000","last_peered":"2024-05-06T00:30:40.725286+0000","last_clean":"2024-05-06T00:30:40.725286+0000","last_became_active":"2024-05-06T00:30:30.568390+0000","last_became_peered":"2024-05-06T00:30:30.568390+0000","last_unstale":"2024-05-06T00:30:40.725286+0000","last_undegraded":"2024-05-06T00:30:40.725286+0000","last_fullsized":"2024-05-06T00:30:40.725286+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:14:14.323344+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.044642163999999998,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13696,"num_read_kb":33605,"num_write":1130,"num_write_kb":8108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"199'407","reported_seq":6998,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725360+0000","last_change":"2024-05-06T00:30:30.665158+0000","last_active":"2024-05-06T00:30:40.725360+0000","last_peered":"2024-05-06T00:30:40.725360+0000","last_clean":"2024-05-06T00:30:40.725360+0000","last_became_active":"2024-05-06T00:30:30.543930+0000","last_became_peered":"2024-05-06T00:30:30.543930+0000","last_unstale":"2024-05-06T00:30:40.725360+0000","last_undegraded":"2024-05-06T00:30:40.725360+0000","last_fullsized":"2024-05-06T00:30:40.725360+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:42:25.940056+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.040220385999999997,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":7280,"num_read_kb":25626,"num_write":414,"num_write_kb":13645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'681","reported_seq":3865,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668064+0000","last_change":"2024-05-06T00:30:30.639087+0000","last_active":"2024-05-06T00:30:40.668064+0000","last_peered":"2024-05-06T00:30:40.668064+0000","last_clean":"2024-05-06T00:30:40.668064+0000","last_became_active":"2024-05-06T00:30:30.560389+0000","last_became_peered":"2024-05-06T00:30:30.560389+0000","last_unstale":"2024-05-06T00:30:40.668064+0000","last_undegraded":"2024-05-06T00:30:40.668064+0000","last_fullsized":"2024-05-06T00:30:40.668064+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:47:13.586263+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.031279479999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5123,"num_read_kb":16860,"num_write":822,"num_write_kb":5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"185'748","reported_seq":14689,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725370+0000","last_change":"2024-05-06T00:30:35.598576+0000","last_active":"2024-05-06T00:30:40.725370+0000","last_peered":"2024-05-06T00:30:40.725370+0000","last_clean":"2024-05-06T00:30:40.725370+0000","last_became_active":"2024-05-06T00:30:35.595815+0000","last_became_peered":"2024-05-06T00:30:35.595815+0000","last_unstale":"2024-05-06T00:30:40.725370+0000","last_undegraded":"2024-05-06T00:30:40.725370+0000","last_fullsized":"2024-05-06T00:30:40.725370+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:45:37.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017909320000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":14404,"num_read_kb":28468,"num_write":531,"num_write_kb":3442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"194'349","reported_seq":4923,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725623+0000","last_change":"2024-05-06T00:30:19.957321+0000","last_active":"2024-05-06T00:30:40.725623+0000","last_peered":"2024-05-06T00:30:40.725623+0000","last_clean":"2024-05-06T00:30:40.725623+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:30:40.725623+0000","last_undegraded":"2024-05-06T00:30:40.725623+0000","last_fullsized":"2024-05-06T00:30:40.725623+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:27:18.137701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003171063,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3664,"num_read_kb":19738,"num_write":297,"num_write_kb":5164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"191'520","reported_seq":2900,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667838+0000","last_change":"2024-05-06T00:30:19.944378+0000","last_active":"2024-05-06T00:30:40.667838+0000","last_peered":"2024-05-06T00:30:40.667838+0000","last_clean":"2024-05-06T00:30:40.667838+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:30:40.667838+0000","last_undegraded":"2024-05-06T00:30:40.667838+0000","last_fullsized":"2024-05-06T00:30:40.667838+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:02:49.118813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004495626,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2722,"num_read_kb":15200,"num_write":616,"num_write_kb":1916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"191'298","reported_seq":5536,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665623+0000","last_change":"2024-05-06T00:30:36.646099+0000","last_active":"2024-05-06T00:30:40.665623+0000","last_peered":"2024-05-06T00:30:40.665623+0000","last_clean":"2024-05-06T00:30:40.665623+0000","last_became_active":"2024-05-06T00:30:36.641378+0000","last_became_peered":"2024-05-06T00:30:36.641378+0000","last_unstale":"2024-05-06T00:30:40.665623+0000","last_undegraded":"2024-05-06T00:30:40.665623+0000","last_fullsized":"2024-05-06T00:30:40.665623+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:03:14.811762+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031319316999999999,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":4880,"num_read_kb":11906,"num_write":264,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":76800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"199'529","reported_seq":7922,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665809+0000","last_change":"2024-05-06T00:30:36.655087+0000","last_active":"2024-05-06T00:30:40.665809+0000","last_peered":"2024-05-06T00:30:40.665809+0000","last_clean":"2024-05-06T00:30:40.665809+0000","last_became_active":"2024-05-06T00:30:36.651695+0000","last_became_peered":"2024-05-06T00:30:36.651695+0000","last_unstale":"2024-05-06T00:30:40.665809+0000","last_undegraded":"2024-05-06T00:30:40.665809+0000","last_fullsized":"2024-05-06T00:30:40.665809+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:30.279994+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.030926555000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":7962,"num_read_kb":31384,"num_write":673,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":237,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666855+0000","last_change":"2024-05-06T00:30:36.653124+0000","last_active":"2024-05-06T00:30:40.666855+0000","last_peered":"2024-05-06T00:30:40.666855+0000","last_clean":"2024-05-06T00:30:40.666855+0000","last_became_active":"2024-05-06T00:30:36.652331+0000","last_became_peered":"2024-05-06T00:30:36.652331+0000","last_unstale":"2024-05-06T00:30:40.666855+0000","last_undegraded":"2024-05-06T00:30:40.666855+0000","last_fullsized":"2024-05-06T00:30:40.666855+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:45:22.047908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"199'1005","reported_seq":7890,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725264+0000","last_change":"2024-05-06T00:30:30.709509+0000","last_active":"2024-05-06T00:30:40.725264+0000","last_peered":"2024-05-06T00:30:40.725264+0000","last_clean":"2024-05-06T00:30:40.725264+0000","last_became_active":"2024-05-06T00:30:30.569873+0000","last_became_peered":"2024-05-06T00:30:30.569873+0000","last_unstale":"2024-05-06T00:30:40.725264+0000","last_undegraded":"2024-05-06T00:30:40.725264+0000","last_fullsized":"2024-05-06T00:30:40.725264+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:37:31.390169+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.024654692999999998,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":10870,"num_read_kb":29913,"num_write":1138,"num_write_kb":7394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668489+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:30:40.668489+0000","last_peered":"2024-05-06T00:30:40.668489+0000","last_clean":"2024-05-06T00:30:40.668489+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:30:40.668489+0000","last_undegraded":"2024-05-06T00:30:40.668489+0000","last_fullsized":"2024-05-06T00:30:40.668489+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"191'766","reported_seq":6743,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725403+0000","last_change":"2024-05-06T00:30:35.593017+0000","last_active":"2024-05-06T00:30:40.725403+0000","last_peered":"2024-05-06T00:30:40.725403+0000","last_clean":"2024-05-06T00:30:40.725403+0000","last_became_active":"2024-05-06T00:30:35.589029+0000","last_became_peered":"2024-05-06T00:30:35.589029+0000","last_unstale":"2024-05-06T00:30:40.725403+0000","last_undegraded":"2024-05-06T00:30:40.725403+0000","last_fullsized":"2024-05-06T00:30:40.725403+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:18:10.559148+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.003902722,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8939,"num_read_kb":19825,"num_write":709,"num_write_kb":2787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666154+0000","last_change":"2024-05-06T00:30:36.646507+0000","last_active":"2024-05-06T00:30:40.666154+0000","last_peered":"2024-05-06T00:30:40.666154+0000","last_clean":"2024-05-06T00:30:40.666154+0000","last_became_active":"2024-05-06T00:30:36.644967+0000","last_became_peered":"2024-05-06T00:30:36.644967+0000","last_unstale":"2024-05-06T00:30:40.666154+0000","last_undegraded":"2024-05-06T00:30:40.666154+0000","last_fullsized":"2024-05-06T00:30:40.666154+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:16:47.890394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"191'657","reported_seq":9441,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668189+0000","last_change":"2024-05-06T00:30:19.945917+0000","last_active":"2024-05-06T00:30:40.668189+0000","last_peered":"2024-05-06T00:30:40.668189+0000","last_clean":"2024-05-06T00:30:40.668189+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:30:40.668189+0000","last_undegraded":"2024-05-06T00:30:40.668189+0000","last_fullsized":"2024-05-06T00:30:40.668189+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:49:12.946087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060800150000000002,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":10036,"num_read_kb":30901,"num_write":694,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668452+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:30:40.668452+0000","last_peered":"2024-05-06T00:30:40.668452+0000","last_clean":"2024-05-06T00:30:40.668452+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:30:40.668452+0000","last_undegraded":"2024-05-06T00:30:40.668452+0000","last_fullsized":"2024-05-06T00:30:40.668452+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"191'559","reported_seq":5683,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667793+0000","last_change":"2024-05-06T00:30:19.941179+0000","last_active":"2024-05-06T00:30:40.667793+0000","last_peered":"2024-05-06T00:30:40.667793+0000","last_clean":"2024-05-06T00:30:40.667793+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:30:40.667793+0000","last_undegraded":"2024-05-06T00:30:40.667793+0000","last_fullsized":"2024-05-06T00:30:40.667793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:38:10.377797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016637620000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":6637,"num_read_kb":16275,"num_write":486,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"199'538","reported_seq":4051,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725423+0000","last_change":"2024-05-06T00:30:30.742622+0000","last_active":"2024-05-06T00:30:40.725423+0000","last_peered":"2024-05-06T00:30:40.725423+0000","last_clean":"2024-05-06T00:30:40.725423+0000","last_became_active":"2024-05-06T00:30:30.561444+0000","last_became_peered":"2024-05-06T00:30:30.561444+0000","last_unstale":"2024-05-06T00:30:40.725423+0000","last_undegraded":"2024-05-06T00:30:40.725423+0000","last_fullsized":"2024-05-06T00:30:40.725423+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:39:30.619891+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.034165040000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":17,"num_read":3499,"num_read_kb":25511,"num_write":548,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":105,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667480+0000","last_change":"2024-05-06T00:30:39.648249+0000","last_active":"2024-05-06T00:30:40.667480+0000","last_peered":"2024-05-06T00:30:40.667480+0000","last_clean":"2024-05-06T00:30:40.667480+0000","last_became_active":"2024-05-06T00:30:39.647436+0000","last_became_peered":"2024-05-06T00:30:39.647436+0000","last_unstale":"2024-05-06T00:30:40.667480+0000","last_undegraded":"2024-05-06T00:30:40.667480+0000","last_fullsized":"2024-05-06T00:30:40.667480+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:15:38.102710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"192'657","reported_seq":6870,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667981+0000","last_change":"2024-05-06T00:30:19.944664+0000","last_active":"2024-05-06T00:30:40.667981+0000","last_peered":"2024-05-06T00:30:40.667981+0000","last_clean":"2024-05-06T00:30:40.667981+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:30:40.667981+0000","last_undegraded":"2024-05-06T00:30:40.667981+0000","last_fullsized":"2024-05-06T00:30:40.667981+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:56:10.880278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051286049999999996,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7148,"num_read_kb":27106,"num_write":776,"num_write_kb":4745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665403+0000","last_change":"2024-05-06T00:30:36.644291+0000","last_active":"2024-05-06T00:30:40.665403+0000","last_peered":"2024-05-06T00:30:40.665403+0000","last_clean":"2024-05-06T00:30:40.665403+0000","last_became_active":"2024-05-06T00:30:36.643456+0000","last_became_peered":"2024-05-06T00:30:36.643456+0000","last_unstale":"2024-05-06T00:30:40.665403+0000","last_undegraded":"2024-05-06T00:30:40.665403+0000","last_fullsized":"2024-05-06T00:30:40.665403+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:25:26.257376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"199'383","reported_seq":4994,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725003+0000","last_change":"2024-05-06T00:30:30.656302+0000","last_active":"2024-05-06T00:30:40.725003+0000","last_peered":"2024-05-06T00:30:40.725003+0000","last_clean":"2024-05-06T00:30:40.725003+0000","last_became_active":"2024-05-06T00:30:30.562288+0000","last_became_peered":"2024-05-06T00:30:30.562288+0000","last_unstale":"2024-05-06T00:30:40.725003+0000","last_undegraded":"2024-05-06T00:30:40.725003+0000","last_fullsized":"2024-05-06T00:30:40.725003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:53:00.473185+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.028356438000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":4034,"num_read_kb":14959,"num_write":372,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665364+0000","last_change":"2024-05-06T00:30:36.643966+0000","last_active":"2024-05-06T00:30:40.665364+0000","last_peered":"2024-05-06T00:30:40.665364+0000","last_clean":"2024-05-06T00:30:40.665364+0000","last_became_active":"2024-05-06T00:30:36.642148+0000","last_became_peered":"2024-05-06T00:30:36.642148+0000","last_unstale":"2024-05-06T00:30:40.665364+0000","last_undegraded":"2024-05-06T00:30:40.665364+0000","last_fullsized":"2024-05-06T00:30:40.665364+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:19:16.556193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"193'778","reported_seq":10467,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668186+0000","last_change":"2024-05-06T00:30:19.947559+0000","last_active":"2024-05-06T00:30:40.668186+0000","last_peered":"2024-05-06T00:30:40.668186+0000","last_clean":"2024-05-06T00:30:40.668186+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:30:40.668186+0000","last_undegraded":"2024-05-06T00:30:40.668186+0000","last_fullsized":"2024-05-06T00:30:40.668186+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:23:03.571190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065619830000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":12206,"num_read_kb":24770,"num_write":877,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667373+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:30:40.667373+0000","last_peered":"2024-05-06T00:30:40.667373+0000","last_clean":"2024-05-06T00:30:40.667373+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:30:40.667373+0000","last_undegraded":"2024-05-06T00:30:40.667373+0000","last_fullsized":"2024-05-06T00:30:40.667373+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":86,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665277+0000","last_change":"2024-05-06T00:30:36.642156+0000","last_active":"2024-05-06T00:30:40.665277+0000","last_peered":"2024-05-06T00:30:40.665277+0000","last_clean":"2024-05-06T00:30:40.665277+0000","last_became_active":"2024-05-06T00:30:36.641260+0000","last_became_peered":"2024-05-06T00:30:36.641260+0000","last_unstale":"2024-05-06T00:30:40.665277+0000","last_undegraded":"2024-05-06T00:30:40.665277+0000","last_fullsized":"2024-05-06T00:30:40.665277+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:31.624557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"191'620","reported_seq":6214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725308+0000","last_change":"2024-05-06T00:30:35.595715+0000","last_active":"2024-05-06T00:30:40.725308+0000","last_peered":"2024-05-06T00:30:40.725308+0000","last_clean":"2024-05-06T00:30:40.725308+0000","last_became_active":"2024-05-06T00:30:35.591860+0000","last_became_peered":"2024-05-06T00:30:35.591860+0000","last_unstale":"2024-05-06T00:30:40.725308+0000","last_undegraded":"2024-05-06T00:30:40.725308+0000","last_fullsized":"2024-05-06T00:30:40.725308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:47:33.295174+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.021797200999999999,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":6539,"num_read_kb":27944,"num_write":612,"num_write_kb":8155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"199'575","reported_seq":5828,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725392+0000","last_change":"2024-05-06T00:30:30.675754+0000","last_active":"2024-05-06T00:30:40.725392+0000","last_peered":"2024-05-06T00:30:40.725392+0000","last_clean":"2024-05-06T00:30:40.725392+0000","last_became_active":"2024-05-06T00:30:30.545607+0000","last_became_peered":"2024-05-06T00:30:30.545607+0000","last_unstale":"2024-05-06T00:30:40.725392+0000","last_undegraded":"2024-05-06T00:30:40.725392+0000","last_fullsized":"2024-05-06T00:30:40.725392+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:54:11.975131+0000","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.036169352000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":5497,"num_read_kb":27780,"num_write":687,"num_write_kb":13301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"199'805","reported_seq":8521,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665648+0000","last_change":"2024-05-06T00:30:36.656283+0000","last_active":"2024-05-06T00:30:40.665648+0000","last_peered":"2024-05-06T00:30:40.665648+0000","last_clean":"2024-05-06T00:30:40.665648+0000","last_became_active":"2024-05-06T00:30:36.651797+0000","last_became_peered":"2024-05-06T00:30:36.651797+0000","last_unstale":"2024-05-06T00:30:40.665648+0000","last_undegraded":"2024-05-06T00:30:40.665648+0000","last_fullsized":"2024-05-06T00:30:40.665648+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:47:48.879526+0000","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0.028632154999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":25,"num_read":9542,"num_read_kb":44861,"num_write":766,"num_write_kb":8347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"187'351","reported_seq":7154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668166+0000","last_change":"2024-05-06T00:30:19.946118+0000","last_active":"2024-05-06T00:30:40.668166+0000","last_peered":"2024-05-06T00:30:40.668166+0000","last_clean":"2024-05-06T00:30:40.668166+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:30:40.668166+0000","last_undegraded":"2024-05-06T00:30:40.668166+0000","last_fullsized":"2024-05-06T00:30:40.668166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:06:25.304781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057155160000000003,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5046,"num_read_kb":27757,"num_write":349,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"188'420","reported_seq":6843,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:39.652878+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:39.650059+0000","last_became_peered":"2024-05-06T00:30:39.650059+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:14:23.610386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057267869999999997,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":6116,"num_read_kb":54371,"num_write":370,"num_write_kb":8906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"187'890","reported_seq":12713,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666252+0000","last_change":"2024-05-06T00:30:36.650116+0000","last_active":"2024-05-06T00:30:40.666252+0000","last_peered":"2024-05-06T00:30:40.666252+0000","last_clean":"2024-05-06T00:30:40.666252+0000","last_became_active":"2024-05-06T00:30:36.645606+0000","last_became_peered":"2024-05-06T00:30:36.645606+0000","last_unstale":"2024-05-06T00:30:40.666252+0000","last_undegraded":"2024-05-06T00:30:40.666252+0000","last_fullsized":"2024-05-06T00:30:40.666252+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:34:38.495321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015459429999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":13504,"num_read_kb":33376,"num_write":1160,"num_write_kb":4653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"199'867","reported_seq":12481,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668014+0000","last_change":"2024-05-06T00:30:39.656310+0000","last_active":"2024-05-06T00:30:40.668014+0000","last_peered":"2024-05-06T00:30:40.668014+0000","last_clean":"2024-05-06T00:30:40.668014+0000","last_became_active":"2024-05-06T00:30:39.652991+0000","last_became_peered":"2024-05-06T00:30:39.652991+0000","last_unstale":"2024-05-06T00:30:40.668014+0000","last_undegraded":"2024-05-06T00:30:40.668014+0000","last_fullsized":"2024-05-06T00:30:40.668014+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:35:14.638153+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.035058351000000001,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":15156,"num_read_kb":31725,"num_write":1055,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"190'348","reported_seq":4259,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668185+0000","last_change":"2024-05-06T00:30:19.943335+0000","last_active":"2024-05-06T00:30:40.668185+0000","last_peered":"2024-05-06T00:30:40.668185+0000","last_clean":"2024-05-06T00:30:40.668185+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:30:40.668185+0000","last_undegraded":"2024-05-06T00:30:40.668185+0000","last_fullsized":"2024-05-06T00:30:40.668185+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:42:14.046101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033172480000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":3496,"num_read_kb":19343,"num_write":377,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'790","reported_seq":11712,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667348+0000","last_change":"2024-05-06T00:30:39.652246+0000","last_active":"2024-05-06T00:30:40.667348+0000","last_peered":"2024-05-06T00:30:40.667348+0000","last_clean":"2024-05-06T00:30:40.667348+0000","last_became_active":"2024-05-06T00:30:39.649412+0000","last_became_peered":"2024-05-06T00:30:39.649412+0000","last_unstale":"2024-05-06T00:30:40.667348+0000","last_undegraded":"2024-05-06T00:30:40.667348+0000","last_fullsized":"2024-05-06T00:30:40.667348+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:51:30.132753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001791927,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13318,"num_read_kb":32027,"num_write":733,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"199'437","reported_seq":5748,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665489+0000","last_change":"2024-05-06T00:30:36.655967+0000","last_active":"2024-05-06T00:30:40.665489+0000","last_peered":"2024-05-06T00:30:40.665489+0000","last_clean":"2024-05-06T00:30:40.665489+0000","last_became_active":"2024-05-06T00:30:36.652383+0000","last_became_peered":"2024-05-06T00:30:36.652383+0000","last_unstale":"2024-05-06T00:30:40.665489+0000","last_undegraded":"2024-05-06T00:30:40.665489+0000","last_fullsized":"2024-05-06T00:30:40.665489+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:56:34.123681+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.029993888999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5408,"num_read_kb":10751,"num_write":542,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"191'477","reported_seq":9126,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725278+0000","last_change":"2024-05-06T00:30:19.956297+0000","last_active":"2024-05-06T00:30:40.725278+0000","last_peered":"2024-05-06T00:30:40.725278+0000","last_clean":"2024-05-06T00:30:40.725278+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:30:40.725278+0000","last_undegraded":"2024-05-06T00:30:40.725278+0000","last_fullsized":"2024-05-06T00:30:40.725278+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:46.269848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002988648,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8695,"num_read_kb":23092,"num_write":389,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"192'673","reported_seq":9486,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665601+0000","last_change":"2024-05-06T00:30:36.651926+0000","last_active":"2024-05-06T00:30:40.665601+0000","last_peered":"2024-05-06T00:30:40.665601+0000","last_clean":"2024-05-06T00:30:40.665601+0000","last_became_active":"2024-05-06T00:30:36.647614+0000","last_became_peered":"2024-05-06T00:30:36.647614+0000","last_unstale":"2024-05-06T00:30:40.665601+0000","last_undegraded":"2024-05-06T00:30:40.665601+0000","last_fullsized":"2024-05-06T00:30:40.665601+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:16:00.581426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00351067,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":10781,"num_read_kb":40700,"num_write":619,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":623,"num_object_clones":0,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":619,"num_read":263747,"num_read_kb":883054,"num_write":20048,"num_write_kb":192626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5734,"num_bytes_recovered":72015234,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},"log_size":19745,"ondisk_log_size":19745,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640313,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60032,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311808,"statfs":{"total":96636764160,"available":96575291392,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":198,"seq":850403524613,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":2476,"kb_used_omap":44,"kb_used_meta":29331,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45664,"internal_metadata":30035360},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640311,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640312,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47544,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324296,"statfs":{"total":96636764160,"available":96588079104,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640313,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47552,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324288,"statfs":{"total":96636764160,"available":96588070912,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640311,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48132,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323708,"statfs":{"total":96636764160,"available":96587476992,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:30:45.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T00:30:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:45.625 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":210,"fsid":"b583c614-b2ca-4769-90c0-44726bf69b34","created":"2024-05-06T00:26:16.923985+0000","modified":"2024-05-06T00:30:40.653120+0000","last_up_change":"2024-05-06T00:30:29.524624+0000","last_in_change":"2024-05-06T00:28:54.626879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T00:26:28.436494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-06T00:26:33.941780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-05-06T00:26:42.622595+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d31b9a6a-864b-4c24-99db-8cc6cf3a5c24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6817","nonce":1427564096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6819","nonce":1427564096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6823","nonce":1427564096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1427564096},{"type":"v1","addr":"172.21.15.78:6821","nonce":1427564096}]},"public_addr":"172.21.15.78:6817/1427564096","cluster_addr":"172.21.15.78:6819/1427564096","heartbeat_back_addr":"172.21.15.78:6823/1427564096","heartbeat_front_addr":"172.21.15.78:6821/1427564096","state":["exists","up"]},{"osd":1,"uuid":"86509826-0e53-4678-8260-17c97c05af54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6801","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6803","nonce":2663808859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6805","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6807","nonce":2663808859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6815","nonce":2663808859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":2663808859},{"type":"v1","addr":"172.21.15.78:6811","nonce":2663808859}]},"public_addr":"172.21.15.78:6803/2663808859","cluster_addr":"172.21.15.78:6807/2663808859","heartbeat_back_addr":"172.21.15.78:6815/2663808859","heartbeat_front_addr":"172.21.15.78:6811/2663808859","state":["exists","up"]},{"osd":2,"uuid":"8973b524-521e-41c0-87f2-85cb390fbfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6802","nonce":385938386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6806","nonce":385938386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6814","nonce":385938386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":385938386},{"type":"v1","addr":"172.21.15.78:6810","nonce":385938386}]},"public_addr":"172.21.15.78:6802/385938386","cluster_addr":"172.21.15.78:6806/385938386","heartbeat_back_addr":"172.21.15.78:6814/385938386","heartbeat_front_addr":"172.21.15.78:6810/385938386","state":["exists","up"]},{"osd":3,"uuid":"3b0d07bd-132e-48ff-83ac-7182eee05df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6812","nonce":4178038317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6816","nonce":4178038317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6822","nonce":4178038317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6817","nonce":4178038317},{"type":"v1","addr":"172.21.15.146:6819","nonce":4178038317}]},"public_addr":"172.21.15.146:6812/4178038317","cluster_addr":"172.21.15.146:6816/4178038317","heartbeat_back_addr":"172.21.15.146:6822/4178038317","heartbeat_front_addr":"172.21.15.146:6819/4178038317","state":["exists","up"]},{"osd":4,"uuid":"0e031319-bb7b-4cf4-af0c-b0aaa79e0bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":174,"up_from":198,"up_thru":198,"down_at":175,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6809","nonce":3162350242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6813","nonce":3162350242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6823","nonce":3162350242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":3162350242},{"type":"v1","addr":"172.21.15.146:6818","nonce":3162350242}]},"public_addr":"172.21.15.146:6809/3162350242","cluster_addr":"172.21.15.146:6813/3162350242","heartbeat_back_addr":"172.21.15.146:6823/3162350242","heartbeat_front_addr":"172.21.15.146:6818/3162350242","state":["exists","up"]},{"osd":5,"uuid":"34b61f5f-e407-4414-8bc5-11263bbe261a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6801","nonce":2019146845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6803","nonce":2019146845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6807","nonce":2019146845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2019146845},{"type":"v1","addr":"172.21.15.146:6805","nonce":2019146845}]},"public_addr":"172.21.15.146:6801/2019146845","cluster_addr":"172.21.15.146:6803/2019146845","heartbeat_back_addr":"172.21.15.146:6807/2019146845","heartbeat_front_addr":"172.21.15.146:6805/2019146845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.276988+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.054473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.154558+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.284746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.259847+0000","dead_epoch":175},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T00:26:24.180375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1","mappings":[{"from":2,"to":3}]},{"pgid":"3.6","mappings":[{"from":1,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T00:30:46.645 INFO:tasks.ceph:Scrubbing osd.0 2024-05-06T00:30:46.646 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-06T00:30:46.810 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-06T00:30:46.810 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:46.826 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-06T00:30:47.114 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 0 to deep-scrub 2024-05-06T00:30:47.133 INFO:tasks.ceph:Scrubbing osd.1 2024-05-06T00:30:47.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-06T00:30:47.279 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:47.279 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-06T00:30:47.279 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:47.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-06T00:30:47.570 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 1 to deep-scrub 2024-05-06T00:30:47.587 INFO:tasks.ceph:Scrubbing osd.2 2024-05-06T00:30:47.587 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-06T00:30:47.720 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:47.720 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-06T00:30:47.720 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:47.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-06T00:30:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 2 to deep-scrub 2024-05-06T00:30:48.023 INFO:tasks.ceph:Scrubbing osd.3 2024-05-06T00:30:48.023 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-06T00:30:48.164 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:48.164 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-06T00:30:48.164 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:48.178 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-06T00:30:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 3 to deep-scrub 2024-05-06T00:30:48.483 INFO:tasks.ceph:Scrubbing osd.4 2024-05-06T00:30:48.483 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-06T00:30:48.629 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-05-06T00:30:48.629 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:48.646 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-06T00:30:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 4 to deep-scrub 2024-05-06T00:30:48.938 INFO:tasks.ceph:Scrubbing osd.5 2024-05-06T00:30:48.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-06T00:30:49.084 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-06T00:30:49.084 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-06T00:30:49.085 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-06T00:30:49.100 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-06T00:30:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 5 to deep-scrub 2024-05-06T00:30:49.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:30:49.728 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:30:49.728 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:30:49.746 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-05-06T00:30:48.462455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":625,"num_object_clones":0,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":619,"num_read":263813,"num_read_kb":883109,"num_write":20149,"num_write_kb":194254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5740,"num_bytes_recovered":73786338,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19817,"ondisk_log_size":19817,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":283248,"kb_used_data":16000,"kb_used_omap":52,"kb_used_meta":267083,"kb_avail":565947792,"statfs":{"total":579820584960,"available":579530539008,"internally_reserved":0,"allocated":16384000,"data_stored":7584694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53637,"internal_metadata":273493627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.821721"},"pg_stats":[{"pgid":"3.1f","version":"194'672","reported_seq":13024,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725565+0000","last_change":"2024-05-06T00:30:19.956846+0000","last_active":"2024-05-06T00:30:40.725565+0000","last_peered":"2024-05-06T00:30:40.725565+0000","last_clean":"2024-05-06T00:30:40.725565+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:30:40.725565+0000","last_undegraded":"2024-05-06T00:30:40.725565+0000","last_fullsized":"2024-05-06T00:30:40.725565+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028964080000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":12460,"num_read_kb":40065,"num_write":528,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"191'367","reported_seq":9956,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725314+0000","last_change":"2024-05-06T00:30:19.973477+0000","last_active":"2024-05-06T00:30:40.725314+0000","last_peered":"2024-05-06T00:30:40.725314+0000","last_clean":"2024-05-06T00:30:40.725314+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:30:40.725314+0000","last_undegraded":"2024-05-06T00:30:40.725314+0000","last_fullsized":"2024-05-06T00:30:40.725314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:47:17.412771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012177069999999999,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":8762,"num_read_kb":50019,"num_write":296,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"199'456","reported_seq":4006,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:30.637866+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:30.558304+0000","last_became_peered":"2024-05-06T00:30:30.558304+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:58:28.298863+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.058990953999999998,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3578,"num_read_kb":12907,"num_write":404,"num_write_kb":3281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"199'918","reported_seq":9351,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667663+0000","last_change":"2024-05-06T00:30:39.656204+0000","last_active":"2024-05-06T00:30:40.667663+0000","last_peered":"2024-05-06T00:30:40.667663+0000","last_clean":"2024-05-06T00:30:40.667663+0000","last_became_active":"2024-05-06T00:30:39.653000+0000","last_became_peered":"2024-05-06T00:30:39.653000+0000","last_unstale":"2024-05-06T00:30:40.667663+0000","last_undegraded":"2024-05-06T00:30:40.667663+0000","last_fullsized":"2024-05-06T00:30:40.667663+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:11:01.551395+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.035914631000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":15,"num_read":12749,"num_read_kb":34289,"num_write":814,"num_write_kb":5275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"199'1204","reported_seq":10122,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725332+0000","last_change":"2024-05-06T00:30:30.704483+0000","last_active":"2024-05-06T00:30:40.725332+0000","last_peered":"2024-05-06T00:30:40.725332+0000","last_clean":"2024-05-06T00:30:40.725332+0000","last_became_active":"2024-05-06T00:30:30.568390+0000","last_became_peered":"2024-05-06T00:30:30.568390+0000","last_unstale":"2024-05-06T00:30:40.725332+0000","last_undegraded":"2024-05-06T00:30:40.725332+0000","last_fullsized":"2024-05-06T00:30:40.725332+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.044642163999999998,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13696,"num_read_kb":33605,"num_write":1130,"num_write_kb":8108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"199'407","reported_seq":6998,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725360+0000","last_change":"2024-05-06T00:30:30.665158+0000","last_active":"2024-05-06T00:30:40.725360+0000","last_peered":"2024-05-06T00:30:40.725360+0000","last_clean":"2024-05-06T00:30:40.725360+0000","last_became_active":"2024-05-06T00:30:30.543930+0000","last_became_peered":"2024-05-06T00:30:30.543930+0000","last_unstale":"2024-05-06T00:30:40.725360+0000","last_undegraded":"2024-05-06T00:30:40.725360+0000","last_fullsized":"2024-05-06T00:30:40.725360+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:42:25.940056+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.040220385999999997,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":7280,"num_read_kb":25626,"num_write":414,"num_write_kb":13645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'681","reported_seq":3865,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668064+0000","last_change":"2024-05-06T00:30:30.639087+0000","last_active":"2024-05-06T00:30:40.668064+0000","last_peered":"2024-05-06T00:30:40.668064+0000","last_clean":"2024-05-06T00:30:40.668064+0000","last_became_active":"2024-05-06T00:30:30.560389+0000","last_became_peered":"2024-05-06T00:30:30.560389+0000","last_unstale":"2024-05-06T00:30:40.668064+0000","last_undegraded":"2024-05-06T00:30:40.668064+0000","last_fullsized":"2024-05-06T00:30:40.668064+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:47:13.586263+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.031279479999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5123,"num_read_kb":16860,"num_write":822,"num_write_kb":5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"185'748","reported_seq":14689,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725370+0000","last_change":"2024-05-06T00:30:35.598576+0000","last_active":"2024-05-06T00:30:40.725370+0000","last_peered":"2024-05-06T00:30:40.725370+0000","last_clean":"2024-05-06T00:30:40.725370+0000","last_became_active":"2024-05-06T00:30:35.595815+0000","last_became_peered":"2024-05-06T00:30:35.595815+0000","last_unstale":"2024-05-06T00:30:40.725370+0000","last_undegraded":"2024-05-06T00:30:40.725370+0000","last_fullsized":"2024-05-06T00:30:40.725370+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:45:37.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017909320000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":14404,"num_read_kb":28468,"num_write":531,"num_write_kb":3442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"194'349","reported_seq":4924,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725701+0000","last_change":"2024-05-06T00:30:19.957321+0000","last_active":"2024-05-06T00:30:40.725701+0000","last_peered":"2024-05-06T00:30:40.725701+0000","last_clean":"2024-05-06T00:30:40.725701+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:30:40.725701+0000","last_undegraded":"2024-05-06T00:30:40.725701+0000","last_fullsized":"2024-05-06T00:30:40.725701+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003171063,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3664,"num_read_kb":19738,"num_write":297,"num_write_kb":5164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"191'520","reported_seq":2900,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667838+0000","last_change":"2024-05-06T00:30:19.944378+0000","last_active":"2024-05-06T00:30:40.667838+0000","last_peered":"2024-05-06T00:30:40.667838+0000","last_clean":"2024-05-06T00:30:40.667838+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:30:40.667838+0000","last_undegraded":"2024-05-06T00:30:40.667838+0000","last_fullsized":"2024-05-06T00:30:40.667838+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:02:49.118813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004495626,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2722,"num_read_kb":15200,"num_write":616,"num_write_kb":1916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"191'298","reported_seq":5536,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665623+0000","last_change":"2024-05-06T00:30:36.646099+0000","last_active":"2024-05-06T00:30:40.665623+0000","last_peered":"2024-05-06T00:30:40.665623+0000","last_clean":"2024-05-06T00:30:40.665623+0000","last_became_active":"2024-05-06T00:30:36.641378+0000","last_became_peered":"2024-05-06T00:30:36.641378+0000","last_unstale":"2024-05-06T00:30:40.665623+0000","last_undegraded":"2024-05-06T00:30:40.665623+0000","last_fullsized":"2024-05-06T00:30:40.665623+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:03:14.811762+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031319316999999999,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":4880,"num_read_kb":11906,"num_write":264,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":76800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"199'529","reported_seq":7922,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665809+0000","last_change":"2024-05-06T00:30:36.655087+0000","last_active":"2024-05-06T00:30:40.665809+0000","last_peered":"2024-05-06T00:30:40.665809+0000","last_clean":"2024-05-06T00:30:40.665809+0000","last_became_active":"2024-05-06T00:30:36.651695+0000","last_became_peered":"2024-05-06T00:30:36.651695+0000","last_unstale":"2024-05-06T00:30:40.665809+0000","last_undegraded":"2024-05-06T00:30:40.665809+0000","last_fullsized":"2024-05-06T00:30:40.665809+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:02:30.279994+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.030926555000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":7962,"num_read_kb":31384,"num_write":673,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":237,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666855+0000","last_change":"2024-05-06T00:30:36.653124+0000","last_active":"2024-05-06T00:30:40.666855+0000","last_peered":"2024-05-06T00:30:40.666855+0000","last_clean":"2024-05-06T00:30:40.666855+0000","last_became_active":"2024-05-06T00:30:36.652331+0000","last_became_peered":"2024-05-06T00:30:36.652331+0000","last_unstale":"2024-05-06T00:30:40.666855+0000","last_undegraded":"2024-05-06T00:30:40.666855+0000","last_fullsized":"2024-05-06T00:30:40.666855+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:45:22.047908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"199'1005","reported_seq":7891,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725321+0000","last_change":"2024-05-06T00:30:30.709509+0000","last_active":"2024-05-06T00:30:40.725321+0000","last_peered":"2024-05-06T00:30:40.725321+0000","last_clean":"2024-05-06T00:30:40.725321+0000","last_became_active":"2024-05-06T00:30:30.569873+0000","last_became_peered":"2024-05-06T00:30:30.569873+0000","last_unstale":"2024-05-06T00:30:40.725321+0000","last_undegraded":"2024-05-06T00:30:40.725321+0000","last_fullsized":"2024-05-06T00:30:40.725321+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.024654692999999998,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":10870,"num_read_kb":29913,"num_write":1138,"num_write_kb":7394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":409,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668489+0000","last_change":"2024-05-06T00:28:27.992554+0000","last_active":"2024-05-06T00:30:40.668489+0000","last_peered":"2024-05-06T00:30:40.668489+0000","last_clean":"2024-05-06T00:30:40.668489+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:30:40.668489+0000","last_undegraded":"2024-05-06T00:30:40.668489+0000","last_fullsized":"2024-05-06T00:30:40.668489+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:09:15.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"191'766","reported_seq":6743,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725403+0000","last_change":"2024-05-06T00:30:35.593017+0000","last_active":"2024-05-06T00:30:40.725403+0000","last_peered":"2024-05-06T00:30:40.725403+0000","last_clean":"2024-05-06T00:30:40.725403+0000","last_became_active":"2024-05-06T00:30:35.589029+0000","last_became_peered":"2024-05-06T00:30:35.589029+0000","last_unstale":"2024-05-06T00:30:40.725403+0000","last_undegraded":"2024-05-06T00:30:40.725403+0000","last_fullsized":"2024-05-06T00:30:40.725403+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:18:10.559148+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.003902722,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8939,"num_read_kb":19825,"num_write":709,"num_write_kb":2787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666154+0000","last_change":"2024-05-06T00:30:36.646507+0000","last_active":"2024-05-06T00:30:40.666154+0000","last_peered":"2024-05-06T00:30:40.666154+0000","last_clean":"2024-05-06T00:30:40.666154+0000","last_became_active":"2024-05-06T00:30:36.644967+0000","last_became_peered":"2024-05-06T00:30:36.644967+0000","last_unstale":"2024-05-06T00:30:40.666154+0000","last_undegraded":"2024-05-06T00:30:40.666154+0000","last_fullsized":"2024-05-06T00:30:40.666154+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:16:47.890394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"191'657","reported_seq":9441,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668189+0000","last_change":"2024-05-06T00:30:19.945917+0000","last_active":"2024-05-06T00:30:40.668189+0000","last_peered":"2024-05-06T00:30:40.668189+0000","last_clean":"2024-05-06T00:30:40.668189+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:30:40.668189+0000","last_undegraded":"2024-05-06T00:30:40.668189+0000","last_fullsized":"2024-05-06T00:30:40.668189+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:49:12.946087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060800150000000002,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":10036,"num_read_kb":30901,"num_write":694,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":410,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668547+0000","last_change":"2024-05-06T00:28:02.802013+0000","last_active":"2024-05-06T00:30:40.668547+0000","last_peered":"2024-05-06T00:30:40.668547+0000","last_clean":"2024-05-06T00:30:40.668547+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:30:40.668547+0000","last_undegraded":"2024-05-06T00:30:40.668547+0000","last_fullsized":"2024-05-06T00:30:40.668547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:40:20.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"191'559","reported_seq":5683,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667793+0000","last_change":"2024-05-06T00:30:19.941179+0000","last_active":"2024-05-06T00:30:40.667793+0000","last_peered":"2024-05-06T00:30:40.667793+0000","last_clean":"2024-05-06T00:30:40.667793+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:30:40.667793+0000","last_undegraded":"2024-05-06T00:30:40.667793+0000","last_fullsized":"2024-05-06T00:30:40.667793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:38:10.377797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016637620000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":6637,"num_read_kb":16275,"num_write":486,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"199'538","reported_seq":4051,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725423+0000","last_change":"2024-05-06T00:30:30.742622+0000","last_active":"2024-05-06T00:30:40.725423+0000","last_peered":"2024-05-06T00:30:40.725423+0000","last_clean":"2024-05-06T00:30:40.725423+0000","last_became_active":"2024-05-06T00:30:30.561444+0000","last_became_peered":"2024-05-06T00:30:30.561444+0000","last_unstale":"2024-05-06T00:30:40.725423+0000","last_undegraded":"2024-05-06T00:30:40.725423+0000","last_fullsized":"2024-05-06T00:30:40.725423+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:39:30.619891+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.034165040000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":17,"num_read":3499,"num_read_kb":25511,"num_write":548,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":105,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667480+0000","last_change":"2024-05-06T00:30:39.648249+0000","last_active":"2024-05-06T00:30:40.667480+0000","last_peered":"2024-05-06T00:30:40.667480+0000","last_clean":"2024-05-06T00:30:40.667480+0000","last_became_active":"2024-05-06T00:30:39.647436+0000","last_became_peered":"2024-05-06T00:30:39.647436+0000","last_unstale":"2024-05-06T00:30:40.667480+0000","last_undegraded":"2024-05-06T00:30:40.667480+0000","last_fullsized":"2024-05-06T00:30:40.667480+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:15:38.102710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"192'657","reported_seq":6871,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668054+0000","last_change":"2024-05-06T00:30:19.944664+0000","last_active":"2024-05-06T00:30:40.668054+0000","last_peered":"2024-05-06T00:30:40.668054+0000","last_clean":"2024-05-06T00:30:40.668054+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:30:40.668054+0000","last_undegraded":"2024-05-06T00:30:40.668054+0000","last_fullsized":"2024-05-06T00:30:40.668054+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:56:10.880278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051286049999999996,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7148,"num_read_kb":27106,"num_write":776,"num_write_kb":4745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665403+0000","last_change":"2024-05-06T00:30:36.644291+0000","last_active":"2024-05-06T00:30:40.665403+0000","last_peered":"2024-05-06T00:30:40.665403+0000","last_clean":"2024-05-06T00:30:40.665403+0000","last_became_active":"2024-05-06T00:30:36.643456+0000","last_became_peered":"2024-05-06T00:30:36.643456+0000","last_unstale":"2024-05-06T00:30:40.665403+0000","last_undegraded":"2024-05-06T00:30:40.665403+0000","last_fullsized":"2024-05-06T00:30:40.665403+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:25:26.257376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"199'383","reported_seq":4995,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725067+0000","last_change":"2024-05-06T00:30:30.656302+0000","last_active":"2024-05-06T00:30:40.725067+0000","last_peered":"2024-05-06T00:30:40.725067+0000","last_clean":"2024-05-06T00:30:40.725067+0000","last_became_active":"2024-05-06T00:30:30.562288+0000","last_became_peered":"2024-05-06T00:30:30.562288+0000","last_unstale":"2024-05-06T00:30:40.725067+0000","last_undegraded":"2024-05-06T00:30:40.725067+0000","last_fullsized":"2024-05-06T00:30:40.725067+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.028356438000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":4034,"num_read_kb":14959,"num_write":372,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665364+0000","last_change":"2024-05-06T00:30:36.643966+0000","last_active":"2024-05-06T00:30:40.665364+0000","last_peered":"2024-05-06T00:30:40.665364+0000","last_clean":"2024-05-06T00:30:40.665364+0000","last_became_active":"2024-05-06T00:30:36.642148+0000","last_became_peered":"2024-05-06T00:30:36.642148+0000","last_unstale":"2024-05-06T00:30:40.665364+0000","last_undegraded":"2024-05-06T00:30:40.665364+0000","last_fullsized":"2024-05-06T00:30:40.665364+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:19:16.556193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"193'778","reported_seq":10467,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668186+0000","last_change":"2024-05-06T00:30:19.947559+0000","last_active":"2024-05-06T00:30:40.668186+0000","last_peered":"2024-05-06T00:30:40.668186+0000","last_clean":"2024-05-06T00:30:40.668186+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:30:40.668186+0000","last_undegraded":"2024-05-06T00:30:40.668186+0000","last_fullsized":"2024-05-06T00:30:40.668186+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:23:03.571190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065619830000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":12206,"num_read_kb":24770,"num_write":877,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667373+0000","last_change":"2024-05-06T00:28:55.661081+0000","last_active":"2024-05-06T00:30:40.667373+0000","last_peered":"2024-05-06T00:30:40.667373+0000","last_clean":"2024-05-06T00:30:40.667373+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:30:40.667373+0000","last_undegraded":"2024-05-06T00:30:40.667373+0000","last_fullsized":"2024-05-06T00:30:40.667373+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:34.609064+0000","last_clean_scrub_stamp":"2024-05-06T00:26:34.609064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:54:33.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":86,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665277+0000","last_change":"2024-05-06T00:30:36.642156+0000","last_active":"2024-05-06T00:30:40.665277+0000","last_peered":"2024-05-06T00:30:40.665277+0000","last_clean":"2024-05-06T00:30:40.665277+0000","last_became_active":"2024-05-06T00:30:36.641260+0000","last_became_peered":"2024-05-06T00:30:36.641260+0000","last_unstale":"2024-05-06T00:30:40.665277+0000","last_undegraded":"2024-05-06T00:30:40.665277+0000","last_fullsized":"2024-05-06T00:30:40.665277+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:28.571492+0000","last_clean_scrub_stamp":"2024-05-06T00:26:28.571492+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:15:31.624557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"191'620","reported_seq":6214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725308+0000","last_change":"2024-05-06T00:30:35.595715+0000","last_active":"2024-05-06T00:30:40.725308+0000","last_peered":"2024-05-06T00:30:40.725308+0000","last_clean":"2024-05-06T00:30:40.725308+0000","last_became_active":"2024-05-06T00:30:35.591860+0000","last_became_peered":"2024-05-06T00:30:35.591860+0000","last_unstale":"2024-05-06T00:30:40.725308+0000","last_undegraded":"2024-05-06T00:30:40.725308+0000","last_fullsized":"2024-05-06T00:30:40.725308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:47:33.295174+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.021797200999999999,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":6539,"num_read_kb":27944,"num_write":612,"num_write_kb":8155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"199'575","reported_seq":5828,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725392+0000","last_change":"2024-05-06T00:30:30.675754+0000","last_active":"2024-05-06T00:30:40.725392+0000","last_peered":"2024-05-06T00:30:40.725392+0000","last_clean":"2024-05-06T00:30:40.725392+0000","last_became_active":"2024-05-06T00:30:30.545607+0000","last_became_peered":"2024-05-06T00:30:30.545607+0000","last_unstale":"2024-05-06T00:30:40.725392+0000","last_undegraded":"2024-05-06T00:30:40.725392+0000","last_fullsized":"2024-05-06T00:30:40.725392+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:54:11.975131+0000","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.036169352000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":5497,"num_read_kb":27780,"num_write":687,"num_write_kb":13301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"199'805","reported_seq":8521,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665648+0000","last_change":"2024-05-06T00:30:36.656283+0000","last_active":"2024-05-06T00:30:40.665648+0000","last_peered":"2024-05-06T00:30:40.665648+0000","last_clean":"2024-05-06T00:30:40.665648+0000","last_became_active":"2024-05-06T00:30:36.651797+0000","last_became_peered":"2024-05-06T00:30:36.651797+0000","last_unstale":"2024-05-06T00:30:40.665648+0000","last_undegraded":"2024-05-06T00:30:40.665648+0000","last_fullsized":"2024-05-06T00:30:40.665648+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:47:48.879526+0000","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0.028632154999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":25,"num_read":9542,"num_read_kb":44861,"num_write":766,"num_write_kb":8347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"187'351","reported_seq":7154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668166+0000","last_change":"2024-05-06T00:30:19.946118+0000","last_active":"2024-05-06T00:30:40.668166+0000","last_peered":"2024-05-06T00:30:40.668166+0000","last_clean":"2024-05-06T00:30:40.668166+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:30:40.668166+0000","last_undegraded":"2024-05-06T00:30:40.668166+0000","last_fullsized":"2024-05-06T00:30:40.668166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:06:25.304781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057155160000000003,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5046,"num_read_kb":27757,"num_write":349,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"188'420","reported_seq":6843,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667817+0000","last_change":"2024-05-06T00:30:39.652878+0000","last_active":"2024-05-06T00:30:40.667817+0000","last_peered":"2024-05-06T00:30:40.667817+0000","last_clean":"2024-05-06T00:30:40.667817+0000","last_became_active":"2024-05-06T00:30:39.650059+0000","last_became_peered":"2024-05-06T00:30:39.650059+0000","last_unstale":"2024-05-06T00:30:40.667817+0000","last_undegraded":"2024-05-06T00:30:40.667817+0000","last_fullsized":"2024-05-06T00:30:40.667817+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:14:23.610386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057267869999999997,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":6116,"num_read_kb":54371,"num_write":370,"num_write_kb":8906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"187'890","reported_seq":12713,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.666252+0000","last_change":"2024-05-06T00:30:36.650116+0000","last_active":"2024-05-06T00:30:40.666252+0000","last_peered":"2024-05-06T00:30:40.666252+0000","last_clean":"2024-05-06T00:30:40.666252+0000","last_became_active":"2024-05-06T00:30:36.645606+0000","last_became_peered":"2024-05-06T00:30:36.645606+0000","last_unstale":"2024-05-06T00:30:40.666252+0000","last_undegraded":"2024-05-06T00:30:40.666252+0000","last_fullsized":"2024-05-06T00:30:40.666252+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:34:38.495321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015459429999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":13504,"num_read_kb":33376,"num_write":1160,"num_write_kb":4653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"199'867","reported_seq":12481,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668014+0000","last_change":"2024-05-06T00:30:39.656310+0000","last_active":"2024-05-06T00:30:40.668014+0000","last_peered":"2024-05-06T00:30:40.668014+0000","last_clean":"2024-05-06T00:30:40.668014+0000","last_became_active":"2024-05-06T00:30:39.652991+0000","last_became_peered":"2024-05-06T00:30:39.652991+0000","last_unstale":"2024-05-06T00:30:40.668014+0000","last_undegraded":"2024-05-06T00:30:40.668014+0000","last_fullsized":"2024-05-06T00:30:40.668014+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:35:14.638153+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.035058351000000001,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":15156,"num_read_kb":31725,"num_write":1055,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"190'348","reported_seq":4259,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.668185+0000","last_change":"2024-05-06T00:30:19.943335+0000","last_active":"2024-05-06T00:30:40.668185+0000","last_peered":"2024-05-06T00:30:40.668185+0000","last_clean":"2024-05-06T00:30:40.668185+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:30:40.668185+0000","last_undegraded":"2024-05-06T00:30:40.668185+0000","last_fullsized":"2024-05-06T00:30:40.668185+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:42:14.046101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033172480000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":3496,"num_read_kb":19343,"num_write":377,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"188'790","reported_seq":11713,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.667497+0000","last_change":"2024-05-06T00:30:39.652246+0000","last_active":"2024-05-06T00:30:40.667497+0000","last_peered":"2024-05-06T00:30:40.667497+0000","last_clean":"2024-05-06T00:30:40.667497+0000","last_became_active":"2024-05-06T00:30:39.649412+0000","last_became_peered":"2024-05-06T00:30:39.649412+0000","last_unstale":"2024-05-06T00:30:40.667497+0000","last_undegraded":"2024-05-06T00:30:40.667497+0000","last_fullsized":"2024-05-06T00:30:40.667497+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:51:30.132753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001791927,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13318,"num_read_kb":32027,"num_write":733,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"199'437","reported_seq":5748,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665489+0000","last_change":"2024-05-06T00:30:36.655967+0000","last_active":"2024-05-06T00:30:40.665489+0000","last_peered":"2024-05-06T00:30:40.665489+0000","last_clean":"2024-05-06T00:30:40.665489+0000","last_became_active":"2024-05-06T00:30:36.652383+0000","last_became_peered":"2024-05-06T00:30:36.652383+0000","last_unstale":"2024-05-06T00:30:40.665489+0000","last_undegraded":"2024-05-06T00:30:40.665489+0000","last_fullsized":"2024-05-06T00:30:40.665489+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:56:34.123681+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.029993888999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5408,"num_read_kb":10751,"num_write":542,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"191'477","reported_seq":9127,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.725329+0000","last_change":"2024-05-06T00:30:19.956297+0000","last_active":"2024-05-06T00:30:40.725329+0000","last_peered":"2024-05-06T00:30:40.725329+0000","last_clean":"2024-05-06T00:30:40.725329+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:30:40.725329+0000","last_undegraded":"2024-05-06T00:30:40.725329+0000","last_fullsized":"2024-05-06T00:30:40.725329+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002988648,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8695,"num_read_kb":23092,"num_write":389,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"192'673","reported_seq":9486,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:40.665601+0000","last_change":"2024-05-06T00:30:36.651926+0000","last_active":"2024-05-06T00:30:40.665601+0000","last_peered":"2024-05-06T00:30:40.665601+0000","last_clean":"2024-05-06T00:30:40.665601+0000","last_became_active":"2024-05-06T00:30:36.647614+0000","last_became_peered":"2024-05-06T00:30:36.647614+0000","last_unstale":"2024-05-06T00:30:40.665601+0000","last_undegraded":"2024-05-06T00:30:40.665601+0000","last_fullsized":"2024-05-06T00:30:40.665601+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:26:42.658790+0000","last_clean_scrub_stamp":"2024-05-06T00:26:42.658790+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:16:00.581426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00351067,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":10781,"num_read_kb":40700,"num_write":619,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":623,"num_object_clones":0,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":619,"num_read":263747,"num_read_kb":883054,"num_write":20048,"num_write_kb":192626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5734,"num_bytes_recovered":72015234,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},"log_size":19745,"ondisk_log_size":19745,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640314,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60032,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311808,"statfs":{"total":96636764160,"available":96575291392,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":198,"seq":850403524613,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":2476,"kb_used_omap":44,"kb_used_meta":29331,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45664,"internal_metadata":30035360},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640312,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640313,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47544,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324296,"statfs":{"total":96636764160,"available":96588079104,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640314,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47552,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324288,"statfs":{"total":96636764160,"available":96588070912,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640312,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48132,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323708,"statfs":{"total":96636764160,"available":96587476992,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:30:49.751 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.751 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.751 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.751 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.752 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.752 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.752 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.752 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.752 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.752 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.752 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.752 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-06T00:26:34.609064+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=34, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-06T00:26:34.609064+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=34, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-06T00:26:34.609064+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=34, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-06T00:26:34.609064+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=34, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.753 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.754 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-06T00:26:34.609064+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=34, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.754 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.754 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-06T00:26:34.609064+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=34, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.754 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.754 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-06T00:26:34.609064+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=34, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.754 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.754 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-06T00:26:34.609064+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=34, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.754 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-06T00:26:28.571492+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=28, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.755 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.755 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.755 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.755 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.755 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.755 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.755 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.755 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.756 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.756 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.756 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.756 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-05-06T00:26:42.658790+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=26, tm_sec=42, tm_wday=0, tm_yday=127, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=6, tm_hour=0, tm_min=30, tm_sec=45, tm_wday=0, tm_yday=127, tm_isdst=0) 2024-05-06T00:30:49.756 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-06T00:31:09.757 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-06T00:31:10.068 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-06T00:31:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-06T00:31:10.086 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-05-06T00:31:08.468817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":625,"num_object_clones":0,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":619,"num_read":263813,"num_read_kb":883109,"num_write":20149,"num_write_kb":194254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5740,"num_bytes_recovered":73786338,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20442,"ondisk_log_size":20442,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":291504,"kb_used_data":16000,"kb_used_omap":52,"kb_used_meta":275403,"kb_avail":565939536,"statfs":{"total":579820584960,"available":579522084864,"internally_reserved":0,"allocated":16384000,"data_stored":7584694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53637,"internal_metadata":282013307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003605"},"pg_stats":[{"pgid":"3.1f","version":"210'691","reported_seq":13051,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:48.671508+0000","last_change":"2024-05-06T00:30:48.671566+0000","last_active":"2024-05-06T00:30:48.671508+0000","last_peered":"2024-05-06T00:30:48.671508+0000","last_clean":"2024-05-06T00:30:48.671508+0000","last_became_active":"2024-05-06T00:28:31.626492+0000","last_became_peered":"2024-05-06T00:28:31.626492+0000","last_unstale":"2024-05-06T00:30:48.671508+0000","last_undegraded":"2024-05-06T00:30:48.671508+0000","last_fullsized":"2024-05-06T00:30:48.671508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'691","last_scrub_stamp":"2024-05-06T00:30:48.671433+0000","last_deep_scrub":"210'691","last_deep_scrub_stamp":"2024-05-06T00:30:48.671433+0000","last_clean_scrub_stamp":"2024-05-06T00:30:48.671433+0000","objects_scrubbed":19,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:00:13.966457+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0028964080000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":12460,"num_read_kb":40065,"num_write":528,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":687644,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"210'383","reported_seq":9982,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:48.300151+0000","last_change":"2024-05-06T00:30:48.300151+0000","last_active":"2024-05-06T00:30:48.300151+0000","last_peered":"2024-05-06T00:30:48.300151+0000","last_clean":"2024-05-06T00:30:48.300151+0000","last_became_active":"2024-05-06T00:28:58.675639+0000","last_became_peered":"2024-05-06T00:28:58.675639+0000","last_unstale":"2024-05-06T00:30:48.300151+0000","last_undegraded":"2024-05-06T00:30:48.300151+0000","last_fullsized":"2024-05-06T00:30:48.300151+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'383","last_scrub_stamp":"2024-05-06T00:30:48.299968+0000","last_deep_scrub":"210'383","last_deep_scrub_stamp":"2024-05-06T00:30:48.299968+0000","last_clean_scrub_stamp":"2024-05-06T00:30:48.299968+0000","objects_scrubbed":16,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:57:23.252706+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0012177069999999999,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":8762,"num_read_kb":50019,"num_write":296,"num_write_kb":18057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":21179516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"210'479","reported_seq":4039,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:53.450830+0000","last_change":"2024-05-06T00:30:53.450830+0000","last_active":"2024-05-06T00:30:53.450830+0000","last_peered":"2024-05-06T00:30:53.450830+0000","last_clean":"2024-05-06T00:30:53.450830+0000","last_became_active":"2024-05-06T00:30:30.558304+0000","last_became_peered":"2024-05-06T00:30:30.558304+0000","last_unstale":"2024-05-06T00:30:53.450830+0000","last_undegraded":"2024-05-06T00:30:53.450830+0000","last_fullsized":"2024-05-06T00:30:53.450830+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'479","last_scrub_stamp":"2024-05-06T00:30:53.450354+0000","last_deep_scrub":"210'479","last_deep_scrub_stamp":"2024-05-06T00:30:53.450354+0000","last_clean_scrub_stamp":"2024-05-06T00:30:53.450354+0000","objects_scrubbed":23,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:19:03.569704+0000","scrub_duration":39,"objects_trimmed":6,"snaptrim_duration":0.058990953999999998,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3578,"num_read_kb":12907,"num_write":404,"num_write_kb":3281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":565760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"210'934","reported_seq":9377,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:57.412778+0000","last_change":"2024-05-06T00:30:57.412778+0000","last_active":"2024-05-06T00:30:57.412778+0000","last_peered":"2024-05-06T00:30:57.412778+0000","last_clean":"2024-05-06T00:30:57.412778+0000","last_became_active":"2024-05-06T00:30:39.653000+0000","last_became_peered":"2024-05-06T00:30:39.653000+0000","last_unstale":"2024-05-06T00:30:57.412778+0000","last_undegraded":"2024-05-06T00:30:57.412778+0000","last_fullsized":"2024-05-06T00:30:57.412778+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'934","last_scrub_stamp":"2024-05-06T00:30:57.412492+0000","last_deep_scrub":"210'934","last_deep_scrub_stamp":"2024-05-06T00:30:57.412492+0000","last_clean_scrub_stamp":"2024-05-06T00:30:57.412492+0000","objects_scrubbed":16,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:16:40.494171+0000","scrub_duration":31,"objects_trimmed":14,"snaptrim_duration":0.035914631000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":15,"num_read":12749,"num_read_kb":34289,"num_write":814,"num_write_kb":5275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":1388650,"num_keys_recovered":136,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"210'1224","reported_seq":10150,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:51.591124+0000","last_change":"2024-05-06T00:30:51.591218+0000","last_active":"2024-05-06T00:30:51.591124+0000","last_peered":"2024-05-06T00:30:51.591124+0000","last_clean":"2024-05-06T00:30:51.591124+0000","last_became_active":"2024-05-06T00:30:30.568390+0000","last_became_peered":"2024-05-06T00:30:30.568390+0000","last_unstale":"2024-05-06T00:30:51.591124+0000","last_undegraded":"2024-05-06T00:30:51.591124+0000","last_fullsized":"2024-05-06T00:30:51.591124+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'1224","last_scrub_stamp":"2024-05-06T00:30:51.590935+0000","last_deep_scrub":"210'1224","last_deep_scrub_stamp":"2024-05-06T00:30:51.590935+0000","last_clean_scrub_stamp":"2024-05-06T00:30:51.590935+0000","objects_scrubbed":20,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:39:50.412617+0000","scrub_duration":43,"objects_trimmed":28,"snaptrim_duration":0.044642163999999998,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13696,"num_read_kb":33605,"num_write":1130,"num_write_kb":8108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":622740,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"210'428","reported_seq":7028,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:49.817093+0000","last_change":"2024-05-06T00:30:49.817208+0000","last_active":"2024-05-06T00:30:49.817093+0000","last_peered":"2024-05-06T00:30:49.817093+0000","last_clean":"2024-05-06T00:30:49.817093+0000","last_became_active":"2024-05-06T00:30:30.543930+0000","last_became_peered":"2024-05-06T00:30:30.543930+0000","last_unstale":"2024-05-06T00:30:49.817093+0000","last_undegraded":"2024-05-06T00:30:49.817093+0000","last_fullsized":"2024-05-06T00:30:49.817093+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'428","last_scrub_stamp":"2024-05-06T00:30:49.816913+0000","last_deep_scrub":"210'428","last_deep_scrub_stamp":"2024-05-06T00:30:49.816913+0000","last_clean_scrub_stamp":"2024-05-06T00:30:49.816913+0000","objects_scrubbed":21,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:59:38.203264+0000","scrub_duration":35,"objects_trimmed":8,"snaptrim_duration":0.040220385999999997,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":7280,"num_read_kb":25626,"num_write":414,"num_write_kb":13645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"210'693","reported_seq":3886,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:53.698602+0000","last_change":"2024-05-06T00:30:53.698685+0000","last_active":"2024-05-06T00:30:53.698602+0000","last_peered":"2024-05-06T00:30:53.698602+0000","last_clean":"2024-05-06T00:30:53.698602+0000","last_became_active":"2024-05-06T00:30:30.560389+0000","last_became_peered":"2024-05-06T00:30:30.560389+0000","last_unstale":"2024-05-06T00:30:53.698602+0000","last_undegraded":"2024-05-06T00:30:53.698602+0000","last_fullsized":"2024-05-06T00:30:53.698602+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'693","last_scrub_stamp":"2024-05-06T00:30:53.698403+0000","last_deep_scrub":"210'693","last_deep_scrub_stamp":"2024-05-06T00:30:53.698403+0000","last_clean_scrub_stamp":"2024-05-06T00:30:53.698403+0000","objects_scrubbed":12,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:12:37.520463+0000","scrub_duration":19,"objects_trimmed":6,"snaptrim_duration":0.031279479999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5123,"num_read_kb":16860,"num_write":822,"num_write_kb":5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"210'761","reported_seq":14711,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:51.259660+0000","last_change":"2024-05-06T00:30:51.259741+0000","last_active":"2024-05-06T00:30:51.259660+0000","last_peered":"2024-05-06T00:30:51.259660+0000","last_clean":"2024-05-06T00:30:51.259660+0000","last_became_active":"2024-05-06T00:30:35.595815+0000","last_became_peered":"2024-05-06T00:30:35.595815+0000","last_unstale":"2024-05-06T00:30:51.259660+0000","last_undegraded":"2024-05-06T00:30:51.259660+0000","last_fullsized":"2024-05-06T00:30:51.259660+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'761","last_scrub_stamp":"2024-05-06T00:30:51.259561+0000","last_deep_scrub":"210'761","last_deep_scrub_stamp":"2024-05-06T00:30:51.259561+0000","last_clean_scrub_stamp":"2024-05-06T00:30:51.259561+0000","objects_scrubbed":13,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T02:45:38.544244+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0017909320000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":14404,"num_read_kb":28468,"num_write":531,"num_write_kb":3442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":1504906,"num_keys_recovered":93,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"210'369","reported_seq":4952,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:47.666220+0000","last_change":"2024-05-06T00:30:47.666320+0000","last_active":"2024-05-06T00:30:47.666220+0000","last_peered":"2024-05-06T00:30:47.666220+0000","last_clean":"2024-05-06T00:30:47.666220+0000","last_became_active":"2024-05-06T00:28:25.969330+0000","last_became_peered":"2024-05-06T00:28:25.969330+0000","last_unstale":"2024-05-06T00:30:47.666220+0000","last_undegraded":"2024-05-06T00:30:47.666220+0000","last_fullsized":"2024-05-06T00:30:47.666220+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'369","last_scrub_stamp":"2024-05-06T00:30:47.666096+0000","last_deep_scrub":"210'369","last_deep_scrub_stamp":"2024-05-06T00:30:47.666096+0000","last_clean_scrub_stamp":"2024-05-06T00:30:47.666096+0000","objects_scrubbed":20,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:55:08.023590+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.003171063,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3664,"num_read_kb":19738,"num_write":297,"num_write_kb":5164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1447592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"210'536","reported_seq":2926,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:49.493956+0000","last_change":"2024-05-06T00:30:49.493956+0000","last_active":"2024-05-06T00:30:49.493956+0000","last_peered":"2024-05-06T00:30:49.493956+0000","last_clean":"2024-05-06T00:30:49.493956+0000","last_became_active":"2024-05-06T00:28:31.620936+0000","last_became_peered":"2024-05-06T00:28:31.620936+0000","last_unstale":"2024-05-06T00:30:49.493956+0000","last_undegraded":"2024-05-06T00:30:49.493956+0000","last_fullsized":"2024-05-06T00:30:49.493956+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'536","last_scrub_stamp":"2024-05-06T00:30:49.493669+0000","last_deep_scrub":"210'536","last_deep_scrub_stamp":"2024-05-06T00:30:49.493669+0000","last_clean_scrub_stamp":"2024-05-06T00:30:49.493669+0000","objects_scrubbed":16,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:17:05.589302+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.004495626,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2722,"num_read_kb":15200,"num_write":616,"num_write_kb":1916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1477392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"210'312","reported_seq":5559,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:51.835956+0000","last_change":"2024-05-06T00:30:51.836028+0000","last_active":"2024-05-06T00:30:51.835956+0000","last_peered":"2024-05-06T00:30:51.835956+0000","last_clean":"2024-05-06T00:30:51.835956+0000","last_became_active":"2024-05-06T00:30:36.641378+0000","last_became_peered":"2024-05-06T00:30:36.641378+0000","last_unstale":"2024-05-06T00:30:51.835956+0000","last_undegraded":"2024-05-06T00:30:51.835956+0000","last_fullsized":"2024-05-06T00:30:51.835956+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'312","last_scrub_stamp":"2024-05-06T00:30:51.835840+0000","last_deep_scrub":"210'312","last_deep_scrub_stamp":"2024-05-06T00:30:51.835840+0000","last_clean_scrub_stamp":"2024-05-06T00:30:51.835840+0000","objects_scrubbed":14,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:52:46.909448+0000","scrub_duration":19,"objects_trimmed":4,"snaptrim_duration":0.031319316999999999,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":4880,"num_read_kb":11906,"num_write":264,"num_write_kb":15357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":76800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"210'548","reported_seq":7950,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:56.900738+0000","last_change":"2024-05-06T00:30:56.900853+0000","last_active":"2024-05-06T00:30:56.900738+0000","last_peered":"2024-05-06T00:30:56.900738+0000","last_clean":"2024-05-06T00:30:56.900738+0000","last_became_active":"2024-05-06T00:30:36.651695+0000","last_became_peered":"2024-05-06T00:30:36.651695+0000","last_unstale":"2024-05-06T00:30:56.900738+0000","last_undegraded":"2024-05-06T00:30:56.900738+0000","last_fullsized":"2024-05-06T00:30:56.900738+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'548","last_scrub_stamp":"2024-05-06T00:30:56.900547+0000","last_deep_scrub":"210'548","last_deep_scrub_stamp":"2024-05-06T00:30:56.900547+0000","last_clean_scrub_stamp":"2024-05-06T00:30:56.900547+0000","objects_scrubbed":19,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:35:14.483263+0000","scrub_duration":35,"objects_trimmed":8,"snaptrim_duration":0.030926555000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":7962,"num_read_kb":31384,"num_write":673,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1568363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:55.877352+0000","last_change":"2024-05-06T00:30:55.877406+0000","last_active":"2024-05-06T00:30:55.877352+0000","last_peered":"2024-05-06T00:30:55.877352+0000","last_clean":"2024-05-06T00:30:55.877352+0000","last_became_active":"2024-05-06T00:30:36.652331+0000","last_became_peered":"2024-05-06T00:30:36.652331+0000","last_unstale":"2024-05-06T00:30:55.877352+0000","last_undegraded":"2024-05-06T00:30:55.877352+0000","last_fullsized":"2024-05-06T00:30:55.877352+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:30:55.877173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:30:55.877173+0000","last_clean_scrub_stamp":"2024-05-06T00:30:55.877173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:23:51.190584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"210'1020","reported_seq":7915,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:52.559030+0000","last_change":"2024-05-06T00:30:52.559113+0000","last_active":"2024-05-06T00:30:52.559030+0000","last_peered":"2024-05-06T00:30:52.559030+0000","last_clean":"2024-05-06T00:30:52.559030+0000","last_became_active":"2024-05-06T00:30:30.569873+0000","last_became_peered":"2024-05-06T00:30:30.569873+0000","last_unstale":"2024-05-06T00:30:52.559030+0000","last_undegraded":"2024-05-06T00:30:52.559030+0000","last_fullsized":"2024-05-06T00:30:52.559030+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'1020","last_scrub_stamp":"2024-05-06T00:30:52.558803+0000","last_deep_scrub":"210'1020","last_deep_scrub_stamp":"2024-05-06T00:30:52.558803+0000","last_clean_scrub_stamp":"2024-05-06T00:30:52.558803+0000","objects_scrubbed":15,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:02:16.340489+0000","scrub_duration":27,"objects_trimmed":8,"snaptrim_duration":0.024654692999999998,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":10870,"num_read_kb":29913,"num_write":1138,"num_write_kb":7394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2459407,"num_keys_recovered":58,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":419,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:50.640780+0000","last_change":"2024-05-06T00:30:50.640780+0000","last_active":"2024-05-06T00:30:50.640780+0000","last_peered":"2024-05-06T00:30:50.640780+0000","last_clean":"2024-05-06T00:30:50.640780+0000","last_became_active":"2024-05-06T00:28:27.991647+0000","last_became_peered":"2024-05-06T00:28:27.991647+0000","last_unstale":"2024-05-06T00:30:50.640780+0000","last_undegraded":"2024-05-06T00:30:50.640780+0000","last_fullsized":"2024-05-06T00:30:50.640780+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:30:50.640540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:30:50.640540+0000","last_clean_scrub_stamp":"2024-05-06T00:30:50.640540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:10:01.120361+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"210'783","reported_seq":6769,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:49.284639+0000","last_change":"2024-05-06T00:30:49.284789+0000","last_active":"2024-05-06T00:30:49.284639+0000","last_peered":"2024-05-06T00:30:49.284639+0000","last_clean":"2024-05-06T00:30:49.284639+0000","last_became_active":"2024-05-06T00:30:35.589029+0000","last_became_peered":"2024-05-06T00:30:35.589029+0000","last_unstale":"2024-05-06T00:30:49.284639+0000","last_undegraded":"2024-05-06T00:30:49.284639+0000","last_fullsized":"2024-05-06T00:30:49.284639+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'783","last_scrub_stamp":"2024-05-06T00:30:49.284471+0000","last_deep_scrub":"210'783","last_deep_scrub_stamp":"2024-05-06T00:30:49.284471+0000","last_clean_scrub_stamp":"2024-05-06T00:30:49.284471+0000","objects_scrubbed":17,"log_size":783,"log_dups_size":0,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T04:52:37.769140+0000","scrub_duration":23,"objects_trimmed":2,"snaptrim_duration":0.003902722,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8939,"num_read_kb":19825,"num_write":709,"num_write_kb":2787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914126,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":83,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:52.864448+0000","last_change":"2024-05-06T00:30:52.864500+0000","last_active":"2024-05-06T00:30:52.864448+0000","last_peered":"2024-05-06T00:30:52.864448+0000","last_clean":"2024-05-06T00:30:52.864448+0000","last_became_active":"2024-05-06T00:30:36.644967+0000","last_became_peered":"2024-05-06T00:30:36.644967+0000","last_unstale":"2024-05-06T00:30:52.864448+0000","last_undegraded":"2024-05-06T00:30:52.864448+0000","last_fullsized":"2024-05-06T00:30:52.864448+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:30:52.864246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:30:52.864246+0000","last_clean_scrub_stamp":"2024-05-06T00:30:52.864246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:11:58.616827+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"210'682","reported_seq":9477,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:51.683399+0000","last_change":"2024-05-06T00:30:51.683399+0000","last_active":"2024-05-06T00:30:51.683399+0000","last_peered":"2024-05-06T00:30:51.683399+0000","last_clean":"2024-05-06T00:30:51.683399+0000","last_became_active":"2024-05-06T00:28:59.668811+0000","last_became_peered":"2024-05-06T00:28:59.668811+0000","last_unstale":"2024-05-06T00:30:51.683399+0000","last_undegraded":"2024-05-06T00:30:51.683399+0000","last_fullsized":"2024-05-06T00:30:51.683399+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'682","last_scrub_stamp":"2024-05-06T00:30:51.683137+0000","last_deep_scrub":"210'682","last_deep_scrub_stamp":"2024-05-06T00:30:51.683137+0000","last_clean_scrub_stamp":"2024-05-06T00:30:51.683137+0000","objects_scrubbed":25,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:02:42.498392+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.0060800150000000002,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":10036,"num_read_kb":30901,"num_write":694,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":934114,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":419,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:47.651098+0000","last_change":"2024-05-06T00:30:47.651132+0000","last_active":"2024-05-06T00:30:47.651098+0000","last_peered":"2024-05-06T00:30:47.651098+0000","last_clean":"2024-05-06T00:30:47.651098+0000","last_became_active":"2024-05-06T00:28:02.801593+0000","last_became_peered":"2024-05-06T00:28:02.801593+0000","last_unstale":"2024-05-06T00:30:47.651098+0000","last_undegraded":"2024-05-06T00:30:47.651098+0000","last_fullsized":"2024-05-06T00:30:47.651098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:30:47.650989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:30:47.650989+0000","last_clean_scrub_stamp":"2024-05-06T00:30:47.650989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:16:39.173903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"210'574","reported_seq":5707,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:52.479021+0000","last_change":"2024-05-06T00:30:52.479097+0000","last_active":"2024-05-06T00:30:52.479021+0000","last_peered":"2024-05-06T00:30:52.479021+0000","last_clean":"2024-05-06T00:30:52.479021+0000","last_became_active":"2024-05-06T00:28:58.677394+0000","last_became_peered":"2024-05-06T00:28:58.677394+0000","last_unstale":"2024-05-06T00:30:52.479021+0000","last_undegraded":"2024-05-06T00:30:52.479021+0000","last_fullsized":"2024-05-06T00:30:52.479021+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'574","last_scrub_stamp":"2024-05-06T00:30:52.478905+0000","last_deep_scrub":"210'574","last_deep_scrub_stamp":"2024-05-06T00:30:52.478905+0000","last_clean_scrub_stamp":"2024-05-06T00:30:52.478905+0000","objects_scrubbed":15,"log_size":574,"log_dups_size":0,"ondisk_log_size":574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:17:08.671000+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0016637620000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":6637,"num_read_kb":16275,"num_write":486,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1702747,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"210'556","reported_seq":4078,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:51.776823+0000","last_change":"2024-05-06T00:30:51.777035+0000","last_active":"2024-05-06T00:30:51.776823+0000","last_peered":"2024-05-06T00:30:51.776823+0000","last_clean":"2024-05-06T00:30:51.776823+0000","last_became_active":"2024-05-06T00:30:30.561444+0000","last_became_peered":"2024-05-06T00:30:30.561444+0000","last_unstale":"2024-05-06T00:30:51.776823+0000","last_undegraded":"2024-05-06T00:30:51.776823+0000","last_fullsized":"2024-05-06T00:30:51.776823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'556","last_scrub_stamp":"2024-05-06T00:30:51.776551+0000","last_deep_scrub":"210'556","last_deep_scrub_stamp":"2024-05-06T00:30:51.776551+0000","last_clean_scrub_stamp":"2024-05-06T00:30:51.776551+0000","objects_scrubbed":18,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:23:06.634765+0000","scrub_duration":27,"objects_trimmed":8,"snaptrim_duration":0.034165040000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":17,"num_read":3499,"num_read_kb":25511,"num_write":548,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":106624,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":114,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:54.361970+0000","last_change":"2024-05-06T00:30:54.362083+0000","last_active":"2024-05-06T00:30:54.361970+0000","last_peered":"2024-05-06T00:30:54.361970+0000","last_clean":"2024-05-06T00:30:54.361970+0000","last_became_active":"2024-05-06T00:30:39.647436+0000","last_became_peered":"2024-05-06T00:30:39.647436+0000","last_unstale":"2024-05-06T00:30:54.361970+0000","last_undegraded":"2024-05-06T00:30:54.361970+0000","last_fullsized":"2024-05-06T00:30:54.361970+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:30:54.361728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:30:54.361728+0000","last_clean_scrub_stamp":"2024-05-06T00:30:54.361728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:19:14.049805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"210'674","reported_seq":6897,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:52.687459+0000","last_change":"2024-05-06T00:30:52.687550+0000","last_active":"2024-05-06T00:30:52.687459+0000","last_peered":"2024-05-06T00:30:52.687459+0000","last_clean":"2024-05-06T00:30:52.687459+0000","last_became_active":"2024-05-06T00:28:59.669094+0000","last_became_peered":"2024-05-06T00:28:59.669094+0000","last_unstale":"2024-05-06T00:30:52.687459+0000","last_undegraded":"2024-05-06T00:30:52.687459+0000","last_fullsized":"2024-05-06T00:30:52.687459+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'674","last_scrub_stamp":"2024-05-06T00:30:52.687266+0000","last_deep_scrub":"210'674","last_deep_scrub_stamp":"2024-05-06T00:30:52.687266+0000","last_clean_scrub_stamp":"2024-05-06T00:30:52.687266+0000","objects_scrubbed":17,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:03:32.948004+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.0051286049999999996,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7148,"num_read_kb":27106,"num_write":776,"num_write_kb":4745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":3634444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:50.839373+0000","last_change":"2024-05-06T00:30:50.839429+0000","last_active":"2024-05-06T00:30:50.839373+0000","last_peered":"2024-05-06T00:30:50.839373+0000","last_clean":"2024-05-06T00:30:50.839373+0000","last_became_active":"2024-05-06T00:30:36.643456+0000","last_became_peered":"2024-05-06T00:30:36.643456+0000","last_unstale":"2024-05-06T00:30:50.839373+0000","last_undegraded":"2024-05-06T00:30:50.839373+0000","last_fullsized":"2024-05-06T00:30:50.839373+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:30:50.839177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:30:50.839177+0000","last_clean_scrub_stamp":"2024-05-06T00:30:50.839177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:34:29.267214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"210'400","reported_seq":5020,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:50.612092+0000","last_change":"2024-05-06T00:30:50.612184+0000","last_active":"2024-05-06T00:30:50.612092+0000","last_peered":"2024-05-06T00:30:50.612092+0000","last_clean":"2024-05-06T00:30:50.612092+0000","last_became_active":"2024-05-06T00:30:30.562288+0000","last_became_peered":"2024-05-06T00:30:30.562288+0000","last_unstale":"2024-05-06T00:30:50.612092+0000","last_undegraded":"2024-05-06T00:30:50.612092+0000","last_fullsized":"2024-05-06T00:30:50.612092+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'400","last_scrub_stamp":"2024-05-06T00:30:50.611881+0000","last_deep_scrub":"210'400","last_deep_scrub_stamp":"2024-05-06T00:30:50.611881+0000","last_clean_scrub_stamp":"2024-05-06T00:30:50.611881+0000","objects_scrubbed":17,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:29:02.552137+0000","scrub_duration":23,"objects_trimmed":8,"snaptrim_duration":0.028356438000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":4034,"num_read_kb":14959,"num_write":372,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":993280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:49.879905+0000","last_change":"2024-05-06T00:30:49.879960+0000","last_active":"2024-05-06T00:30:49.879905+0000","last_peered":"2024-05-06T00:30:49.879905+0000","last_clean":"2024-05-06T00:30:49.879905+0000","last_became_active":"2024-05-06T00:30:36.642148+0000","last_became_peered":"2024-05-06T00:30:36.642148+0000","last_unstale":"2024-05-06T00:30:49.879905+0000","last_undegraded":"2024-05-06T00:30:49.879905+0000","last_fullsized":"2024-05-06T00:30:49.879905+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:30:49.879711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:30:49.879711+0000","last_clean_scrub_stamp":"2024-05-06T00:30:49.879711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T06:42:03.813705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"210'795","reported_seq":10494,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:49.669451+0000","last_change":"2024-05-06T00:30:49.669451+0000","last_active":"2024-05-06T00:30:49.669451+0000","last_peered":"2024-05-06T00:30:49.669451+0000","last_clean":"2024-05-06T00:30:49.669451+0000","last_became_active":"2024-05-06T00:28:25.969816+0000","last_became_peered":"2024-05-06T00:28:25.969816+0000","last_unstale":"2024-05-06T00:30:49.669451+0000","last_undegraded":"2024-05-06T00:30:49.669451+0000","last_fullsized":"2024-05-06T00:30:49.669451+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'795","last_scrub_stamp":"2024-05-06T00:30:49.669136+0000","last_deep_scrub":"210'795","last_deep_scrub_stamp":"2024-05-06T00:30:49.669136+0000","last_clean_scrub_stamp":"2024-05-06T00:30:49.669136+0000","objects_scrubbed":17,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T03:14:21.531692+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.0065619830000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":12206,"num_read_kb":24770,"num_write":877,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":291038,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":255,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:51.458578+0000","last_change":"2024-05-06T00:30:51.458637+0000","last_active":"2024-05-06T00:30:51.458578+0000","last_peered":"2024-05-06T00:30:51.458578+0000","last_clean":"2024-05-06T00:30:51.458578+0000","last_became_active":"2024-05-06T00:28:55.660352+0000","last_became_peered":"2024-05-06T00:28:55.660352+0000","last_unstale":"2024-05-06T00:30:51.458578+0000","last_undegraded":"2024-05-06T00:30:51.458578+0000","last_fullsized":"2024-05-06T00:30:51.458578+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T00:30:51.458364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T00:30:51.458364+0000","last_clean_scrub_stamp":"2024-05-06T00:30:51.458364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:23:42.137320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"210'74","reported_seq":97,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:48.873901+0000","last_change":"2024-05-06T00:30:48.873975+0000","last_active":"2024-05-06T00:30:48.873901+0000","last_peered":"2024-05-06T00:30:48.873901+0000","last_clean":"2024-05-06T00:30:48.873901+0000","last_became_active":"2024-05-06T00:30:36.641260+0000","last_became_peered":"2024-05-06T00:30:36.641260+0000","last_unstale":"2024-05-06T00:30:48.873901+0000","last_undegraded":"2024-05-06T00:30:48.873901+0000","last_fullsized":"2024-05-06T00:30:48.873901+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'74","last_scrub_stamp":"2024-05-06T00:30:48.873709+0000","last_deep_scrub":"210'74","last_deep_scrub_stamp":"2024-05-06T00:30:48.873709+0000","last_clean_scrub_stamp":"2024-05-06T00:30:48.873709+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:18:52.582807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"210'644","reported_seq":6247,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:50.305909+0000","last_change":"2024-05-06T00:30:50.306000+0000","last_active":"2024-05-06T00:30:50.305909+0000","last_peered":"2024-05-06T00:30:50.305909+0000","last_clean":"2024-05-06T00:30:50.305909+0000","last_became_active":"2024-05-06T00:30:35.591860+0000","last_became_peered":"2024-05-06T00:30:35.591860+0000","last_unstale":"2024-05-06T00:30:50.305909+0000","last_undegraded":"2024-05-06T00:30:50.305909+0000","last_fullsized":"2024-05-06T00:30:50.305909+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'644","last_scrub_stamp":"2024-05-06T00:30:50.305766+0000","last_deep_scrub":"210'644","last_deep_scrub_stamp":"2024-05-06T00:30:50.305766+0000","last_clean_scrub_stamp":"2024-05-06T00:30:50.305766+0000","objects_scrubbed":24,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:37:25.793281+0000","scrub_duration":31,"objects_trimmed":2,"snaptrim_duration":0.021797200999999999,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":6539,"num_read_kb":27944,"num_write":612,"num_write_kb":8155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2762142,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"210'596","reported_seq":5858,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:50.804944+0000","last_change":"2024-05-06T00:30:50.805042+0000","last_active":"2024-05-06T00:30:50.804944+0000","last_peered":"2024-05-06T00:30:50.804944+0000","last_clean":"2024-05-06T00:30:50.804944+0000","last_became_active":"2024-05-06T00:30:30.545607+0000","last_became_peered":"2024-05-06T00:30:30.545607+0000","last_unstale":"2024-05-06T00:30:50.804944+0000","last_undegraded":"2024-05-06T00:30:50.804944+0000","last_fullsized":"2024-05-06T00:30:50.804944+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'596","last_scrub_stamp":"2024-05-06T00:30:50.804830+0000","last_deep_scrub":"210'596","last_deep_scrub_stamp":"2024-05-06T00:30:50.804830+0000","last_clean_scrub_stamp":"2024-05-06T00:30:50.804830+0000","objects_scrubbed":21,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T01:35:27.899935+0000","scrub_duration":31,"objects_trimmed":12,"snaptrim_duration":0.036169352000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":5497,"num_read_kb":27780,"num_write":687,"num_write_kb":13301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2501116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"210'831","reported_seq":8557,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:58.886345+0000","last_change":"2024-05-06T00:30:58.886420+0000","last_active":"2024-05-06T00:30:58.886345+0000","last_peered":"2024-05-06T00:30:58.886345+0000","last_clean":"2024-05-06T00:30:58.886345+0000","last_became_active":"2024-05-06T00:30:36.651797+0000","last_became_peered":"2024-05-06T00:30:36.651797+0000","last_unstale":"2024-05-06T00:30:58.886345+0000","last_undegraded":"2024-05-06T00:30:58.886345+0000","last_fullsized":"2024-05-06T00:30:58.886345+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'831","last_scrub_stamp":"2024-05-06T00:30:58.886146+0000","last_deep_scrub":"210'831","last_deep_scrub_stamp":"2024-05-06T00:30:58.886146+0000","last_clean_scrub_stamp":"2024-05-06T00:30:58.886146+0000","objects_scrubbed":26,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T11:14:27.977471+0000","scrub_duration":55,"objects_trimmed":16,"snaptrim_duration":0.028632154999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":25,"num_read":9542,"num_read_kb":44861,"num_write":766,"num_write_kb":8347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":996876,"num_keys_recovered":49,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"210'371","reported_seq":7183,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:48.640064+0000","last_change":"2024-05-06T00:30:48.640124+0000","last_active":"2024-05-06T00:30:48.640064+0000","last_peered":"2024-05-06T00:30:48.640064+0000","last_clean":"2024-05-06T00:30:48.640064+0000","last_became_active":"2024-05-06T00:28:06.812491+0000","last_became_peered":"2024-05-06T00:28:06.812491+0000","last_unstale":"2024-05-06T00:30:48.640064+0000","last_undegraded":"2024-05-06T00:30:48.640064+0000","last_fullsized":"2024-05-06T00:30:48.640064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'371","last_scrub_stamp":"2024-05-06T00:30:48.639987+0000","last_deep_scrub":"210'371","last_deep_scrub_stamp":"2024-05-06T00:30:48.639987+0000","last_clean_scrub_stamp":"2024-05-06T00:30:48.639987+0000","objects_scrubbed":20,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T07:59:39.714201+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0057155160000000003,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5046,"num_read_kb":27757,"num_write":349,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2147279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"210'443","reported_seq":6875,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:56.444247+0000","last_change":"2024-05-06T00:30:56.444366+0000","last_active":"2024-05-06T00:30:56.444247+0000","last_peered":"2024-05-06T00:30:56.444247+0000","last_clean":"2024-05-06T00:30:56.444247+0000","last_became_active":"2024-05-06T00:30:39.650059+0000","last_became_peered":"2024-05-06T00:30:39.650059+0000","last_unstale":"2024-05-06T00:30:56.444247+0000","last_undegraded":"2024-05-06T00:30:56.444247+0000","last_fullsized":"2024-05-06T00:30:56.444247+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'443","last_scrub_stamp":"2024-05-06T00:30:56.444048+0000","last_deep_scrub":"210'443","last_deep_scrub_stamp":"2024-05-06T00:30:56.444048+0000","last_clean_scrub_stamp":"2024-05-06T00:30:56.444048+0000","objects_scrubbed":23,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:54:14.441053+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.0057267869999999997,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":6116,"num_read_kb":54371,"num_write":370,"num_write_kb":8906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":524534,"num_keys_recovered":55,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"210'918","reported_seq":12751,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:53.883310+0000","last_change":"2024-05-06T00:30:53.883382+0000","last_active":"2024-05-06T00:30:53.883310+0000","last_peered":"2024-05-06T00:30:53.883310+0000","last_clean":"2024-05-06T00:30:53.883310+0000","last_became_active":"2024-05-06T00:30:36.645606+0000","last_became_peered":"2024-05-06T00:30:36.645606+0000","last_unstale":"2024-05-06T00:30:53.883310+0000","last_undegraded":"2024-05-06T00:30:53.883310+0000","last_fullsized":"2024-05-06T00:30:53.883310+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'918","last_scrub_stamp":"2024-05-06T00:30:53.883108+0000","last_deep_scrub":"210'918","last_deep_scrub_stamp":"2024-05-06T00:30:53.883108+0000","last_clean_scrub_stamp":"2024-05-06T00:30:53.883108+0000","objects_scrubbed":28,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:59:13.312095+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.0015459429999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":13504,"num_read_kb":33376,"num_write":1160,"num_write_kb":4653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1961764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"210'891","reported_seq":12514,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:58.389449+0000","last_change":"2024-05-06T00:30:58.389559+0000","last_active":"2024-05-06T00:30:58.389449+0000","last_peered":"2024-05-06T00:30:58.389449+0000","last_clean":"2024-05-06T00:30:58.389449+0000","last_became_active":"2024-05-06T00:30:39.652991+0000","last_became_peered":"2024-05-06T00:30:39.652991+0000","last_unstale":"2024-05-06T00:30:58.389449+0000","last_undegraded":"2024-05-06T00:30:58.389449+0000","last_fullsized":"2024-05-06T00:30:58.389449+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'891","last_scrub_stamp":"2024-05-06T00:30:58.389234+0000","last_deep_scrub":"210'891","last_deep_scrub_stamp":"2024-05-06T00:30:58.389234+0000","last_clean_scrub_stamp":"2024-05-06T00:30:58.389234+0000","objects_scrubbed":24,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T00:34:07.952981+0000","scrub_duration":51,"objects_trimmed":6,"snaptrim_duration":0.035058351000000001,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":24,"num_read":15156,"num_read_kb":31725,"num_write":1055,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3538172,"num_keys_recovered":18,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"210'366","reported_seq":4286,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:50.508741+0000","last_change":"2024-05-06T00:30:50.508836+0000","last_active":"2024-05-06T00:30:50.508741+0000","last_peered":"2024-05-06T00:30:50.508741+0000","last_clean":"2024-05-06T00:30:50.508741+0000","last_became_active":"2024-05-06T00:28:32.627716+0000","last_became_peered":"2024-05-06T00:28:32.627716+0000","last_unstale":"2024-05-06T00:30:50.508741+0000","last_undegraded":"2024-05-06T00:30:50.508741+0000","last_fullsized":"2024-05-06T00:30:50.508741+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'366","last_scrub_stamp":"2024-05-06T00:30:50.508538+0000","last_deep_scrub":"210'366","last_deep_scrub_stamp":"2024-05-06T00:30:50.508538+0000","last_clean_scrub_stamp":"2024-05-06T00:30:50.508538+0000","objects_scrubbed":18,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T05:30:22.966867+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.0033172480000000001,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":3496,"num_read_kb":19343,"num_write":377,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1390068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"210'810","reported_seq":11742,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:55.433128+0000","last_change":"2024-05-06T00:30:55.433225+0000","last_active":"2024-05-06T00:30:55.433128+0000","last_peered":"2024-05-06T00:30:55.433128+0000","last_clean":"2024-05-06T00:30:55.433128+0000","last_became_active":"2024-05-06T00:30:39.649412+0000","last_became_peered":"2024-05-06T00:30:39.649412+0000","last_unstale":"2024-05-06T00:30:55.433128+0000","last_undegraded":"2024-05-06T00:30:55.433128+0000","last_fullsized":"2024-05-06T00:30:55.433128+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'810","last_scrub_stamp":"2024-05-06T00:30:55.432937+0000","last_deep_scrub":"210'810","last_deep_scrub_stamp":"2024-05-06T00:30:55.432937+0000","last_clean_scrub_stamp":"2024-05-06T00:30:55.432937+0000","objects_scrubbed":20,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T10:26:59.797787+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001791927,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":13318,"num_read_kb":32027,"num_write":733,"num_write_kb":4634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1226784,"num_keys_recovered":104,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"210'463","reported_seq":5784,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:57.865568+0000","last_change":"2024-05-06T00:30:57.865637+0000","last_active":"2024-05-06T00:30:57.865568+0000","last_peered":"2024-05-06T00:30:57.865568+0000","last_clean":"2024-05-06T00:30:57.865568+0000","last_became_active":"2024-05-06T00:30:36.652383+0000","last_became_peered":"2024-05-06T00:30:36.652383+0000","last_unstale":"2024-05-06T00:30:57.865568+0000","last_undegraded":"2024-05-06T00:30:57.865568+0000","last_fullsized":"2024-05-06T00:30:57.865568+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'463","last_scrub_stamp":"2024-05-06T00:30:57.865367+0000","last_deep_scrub":"210'463","last_deep_scrub_stamp":"2024-05-06T00:30:57.865367+0000","last_clean_scrub_stamp":"2024-05-06T00:30:57.865367+0000","objects_scrubbed":26,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T09:58:22.380221+0000","scrub_duration":47,"objects_trimmed":6,"snaptrim_duration":0.029993888999999999,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5408,"num_read_kb":10751,"num_write":542,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"210'506","reported_seq":9165,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:49.675100+0000","last_change":"2024-05-06T00:30:49.675264+0000","last_active":"2024-05-06T00:30:49.675100+0000","last_peered":"2024-05-06T00:30:49.675100+0000","last_clean":"2024-05-06T00:30:49.675100+0000","last_became_active":"2024-05-06T00:28:31.626334+0000","last_became_peered":"2024-05-06T00:28:31.626334+0000","last_unstale":"2024-05-06T00:30:49.675100+0000","last_undegraded":"2024-05-06T00:30:49.675100+0000","last_fullsized":"2024-05-06T00:30:49.675100+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'506","last_scrub_stamp":"2024-05-06T00:30:49.674896+0000","last_deep_scrub":"210'506","last_deep_scrub_stamp":"2024-05-06T00:30:49.674896+0000","last_clean_scrub_stamp":"2024-05-06T00:30:49.674896+0000","objects_scrubbed":29,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T08:57:45.528420+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.002988648,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8695,"num_read_kb":23092,"num_write":389,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":2477056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"210'687","reported_seq":9509,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-06T00:30:54.867242+0000","last_change":"2024-05-06T00:30:54.867325+0000","last_active":"2024-05-06T00:30:54.867242+0000","last_peered":"2024-05-06T00:30:54.867242+0000","last_clean":"2024-05-06T00:30:54.867242+0000","last_became_active":"2024-05-06T00:30:36.647614+0000","last_became_peered":"2024-05-06T00:30:36.647614+0000","last_unstale":"2024-05-06T00:30:54.867242+0000","last_undegraded":"2024-05-06T00:30:54.867242+0000","last_fullsized":"2024-05-06T00:30:54.867242+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'687","last_scrub_stamp":"2024-05-06T00:30:54.867050+0000","last_deep_scrub":"210'687","last_deep_scrub_stamp":"2024-05-06T00:30:54.867050+0000","last_clean_scrub_stamp":"2024-05-06T00:30:54.867050+0000","objects_scrubbed":14,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T12:01:15.180001+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.00351067,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":10781,"num_read_kb":40700,"num_write":619,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":8877444,"num_keys_recovered":92,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":623,"num_object_clones":0,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":619,"num_read":263747,"num_read_kb":883054,"num_write":20048,"num_write_kb":192626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5734,"num_bytes_recovered":72015234,"num_keys_recovered":904,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},"log_size":20368,"ondisk_log_size":20368,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640318,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60016,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311824,"statfs":{"total":96636764160,"available":96575307776,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":198,"seq":850403524617,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36020,"kb_used_data":2476,"kb_used_omap":44,"kb_used_meta":33491,"kb_avail":94335820,"statfs":{"total":96636764160,"available":96599879680,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45664,"internal_metadata":34295200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640316,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48116,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323724,"statfs":{"total":96636764160,"available":96587493376,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640317,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47536,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324304,"statfs":{"total":96636764160,"available":96588087296,"internally_reserved":0,"allocated":2531328,"data_stored":1067320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640319,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47540,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324300,"statfs":{"total":96636764160,"available":96588083200,"internally_reserved":0,"allocated":2535424,"data_stored":1067339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640316,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52276,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319564,"statfs":{"total":96636764160,"available":96583233536,"internally_reserved":0,"allocated":3125248,"data_stored":1657688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21366,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T00:31:10.090 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-06T00:31:10.404 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-06T00:31:10.405 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-06T00:31:10.405 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-06T00:31:10.405 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:10.500 INFO:tasks.ceph.osd.0:Stopped 2024-05-06T00:31:10.500 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-06T00:31:10.500 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:10.589 INFO:tasks.ceph.osd.1:Stopped 2024-05-06T00:31:10.589 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-06T00:31:10.590 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:10.681 INFO:tasks.ceph.osd.2:Stopped 2024-05-06T00:31:10.681 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-06T00:31:10.681 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:10.778 INFO:tasks.ceph.osd.3:Stopped 2024-05-06T00:31:10.778 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-06T00:31:10.778 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:10.856 INFO:tasks.ceph.osd.4:Stopped 2024-05-06T00:31:10.856 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-06T00:31:10.856 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:14.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-05-06T00:31:16.958 INFO:tasks.ceph.osd.5:Stopped 2024-05-06T00:31:16.959 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-06T00:31:16.959 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-06T00:31:16.959 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:16.989 INFO:tasks.ceph.mgr.x:Stopped 2024-05-06T00:31:16.989 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-06T00:31:16.989 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-06T00:31:16.989 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:17.014 INFO:tasks.ceph.mon.a:Stopped 2024-05-06T00:31:17.014 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-06T00:31:17.015 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:17.078 INFO:tasks.ceph.mon.c:Stopped 2024-05-06T00:31:17.079 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-06T00:31:17.079 INFO:teuthology.orchestra.run:waiting for 300 2024-05-06T00:31:17.093 INFO:tasks.ceph.mon.b:Stopped 2024-05-06T00:31:17.093 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-06T00:31:17.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-05-06T00:31:17.138 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:31:17.138 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-06T00:31:17.429 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:31:17.430 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-06T00:31:17.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:31:17.609 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-06T00:31:17.810 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:31:17.811 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-06T00:31:18.077 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:31:18.077 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-06T00:31:18.267 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:31:18.267 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-06T00:31:18.490 INFO:tasks.ceph:Archiving mon data... 2024-05-06T00:31:18.491 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/data/mon.a.tgz 2024-05-06T00:31:18.491 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-05-06T00:31:18.497 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.e79DFtCkwx 2024-05-06T00:31:18.497 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.e79DFtCkwx 2024-05-06T00:31:18.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.e79DFtCkwx 2024-05-06T00:31:18.712 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.e79DFtCkwx is 388KB 2024-05-06T00:31:18.810 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.e79DFtCkwx 2024-05-06T00:31:18.815 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/data/mon.c.tgz 2024-05-06T00:31:18.816 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-05-06T00:31:18.865 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.A76th3TPdy 2024-05-06T00:31:18.865 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.A76th3TPdy 2024-05-06T00:31:19.042 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.A76th3TPdy 2024-05-06T00:31:19.104 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.A76th3TPdy is 516KB 2024-05-06T00:31:19.166 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.A76th3TPdy 2024-05-06T00:31:19.171 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/data/mon.b.tgz 2024-05-06T00:31:19.171 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-05-06T00:31:19.177 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.asacZzrTgx 2024-05-06T00:31:19.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.asacZzrTgx 2024-05-06T00:31:19.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-05-06T00:31:19.351 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.asacZzrTgx 2024-05-06T00:31:19.416 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.asacZzrTgx is 516KB 2024-05-06T00:31:19.479 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.asacZzrTgx 2024-05-06T00:31:19.484 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-06T00:31:19.484 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-06T00:31:19.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-06T00:31:19.529 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-06T00:31:19.549 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-06T00:31:19.589 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-06T00:31:19.594 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-06T00:31:19.594 INFO:tasks.ceph:Archiving crash dumps... 2024-05-06T00:31:19.595 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi078/crash 2024-05-06T00:31:19.596 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-06T00:31:19.609 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi146/crash 2024-05-06T00:31:19.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-06T00:31:19.645 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi186/crash 2024-05-06T00:31:19.645 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-06T00:31:19.659 INFO:tasks.ceph:Compressing logs... 2024-05-06T00:31:19.659 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T00:31:19.661 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T00:31:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40358.log 2024-05-06T00:31:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39725.log 2024-05-06T00:31:19.684 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36228.log 2024-05-06T00:31:19.684 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40358.log.gz 2024-05-06T00:31:19.684 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39725.log.gzgzip 2024-05-06T00:31:19.684 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39054.log 2024-05-06T00:31:19.685 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40020.log 2024-05-06T00:31:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36228.log: /var/log/ceph/ceph-client.admin.39054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36228.log.gz 2024-05-06T00:31:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37851.log 2024-05-06T00:31:19.686 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39054.log.gz 2024-05-06T00:31:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2024-05-06T00:31:19.686 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40020.log.gz 2024-05-06T00:31:19.688 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37333.log 2024-05-06T00:31:19.688 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37851.log.gz 2024-05-06T00:31:19.688 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30828.log 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37333.log.gz 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36508.log 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30828.log.gz 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37210.log 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.36508.log: /var/log/ceph/ceph-client.admin.39111.log 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36508.log.gz 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37210.log.gz 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33395.log 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-05-06T00:31:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40072.log 2024-05-06T00:31:19.695 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36658.log 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33395.log.gz 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40072.log.gz 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36149.log 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36658.log.gz 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31583.log 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39526.log 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36149.log.gz 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31583.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.31583.log.gz --verbose 2024-05-06T00:31:19.704 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.33057.log 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39526.log.gz 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36573.log 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39312.log 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33057.log.gz 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36573.log.gz 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30281.log 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39312.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.39312.log.gz 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.33577.log 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30281.log.gz 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33577.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.33577.log.gz 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.30799.log 2024-05-06T00:31:19.705 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39870.log 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38519.log 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30799.log.gz 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39870.log.gz 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31920.log 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38519.log.gz 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33832.log 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28733.log 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31920.log.gz 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33832.log.gz 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37621.log 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28733.log.gz 2024-05-06T00:31:19.706 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32774.log 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32729.log 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37621.log.gz 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32774.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32774.log.gz -5 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32729.log.gz 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36707.log 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37946.log 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36707.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.36707.log.gz -- 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.37978.log 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39024.log 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37946.log.gz 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37978.log.gz 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log 2024-05-06T00:31:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38612.log 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39024.log: /var/log/ceph/ceph-client.admin.39343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39024.log.gz 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34304.log 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38655.log 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38612.log.gz 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34304.log.gz 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31630.log 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38655.log: -- /var/log/ceph/ceph-client.admin.38770.log 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38655.log.gz 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31630.log.gz 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39552.log 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34138.log 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38770.log.gz 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39552.log.gz 2024-05-06T00:31:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36406.log 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36894.log 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34138.log.gz 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36406.log.gz 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38222.log 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36894.log.gz 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38222.log.gz 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39728.log 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29233.log 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39728.log.gz 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35672.log 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34341.log 2024-05-06T00:31:19.709 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29233.log.gz 2024-05-06T00:31:19.728 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23914.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30776.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.23914.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23914.log.gz -5 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/ceph-client.admin.24002.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/objectstore_tool.30776.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30581.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.24002.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24002.log.gz 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/objectstore_tool.30581.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.30776.log.gz 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24026.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30983.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.24026.log: 94.5% -- replaced with /var/log/ceph/objectstore_tool.30581.log.gz 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24026.log.gz -5 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/objectstore_tool.31744.log 2024-05-06T00:31:19.729 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/objectstore_tool.30983.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/objectstore_tool.31744.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23978.log 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23954.log 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.23978.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.23978.log.gz 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.23954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23890.log 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.audit.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23954.log.gz 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.23890.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph-client.admin.23890.log.gz 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1%/var/log/ceph/ceph-osd.4.log: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr: 94.9% -- replaced with /var/log/ceph/objectstore_tool.30983.log.gz 2024-05-06T00:31:19.730 INFO:teuthology.orchestra.run.smithi146.stderr: 95.0% -- replaced with /var/log/ceph/objectstore_tool.31744.log.gz 2024-05-06T00:31:19.769 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35672.log.gz 2024-05-06T00:31:19.769 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20298.log 2024-05-06T00:31:19.769 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31837.log 2024-05-06T00:31:19.769 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34341.log.gz 2024-05-06T00:31:19.769 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.tmp-client.admin.20298.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20298.log.gz 2024-05-06T00:31:19.769 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34179.log 2024-05-06T00:31:19.769 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38022.log 2024-05-06T00:31:19.769 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31837.log.gz 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34179.log.gz 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31737.log 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31503.log 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38022.log.gz 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31737.log.gz 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28910.log 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34278.log 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31503.log.gz 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28910.log.gz 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37264.log 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31710.log 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34278.log.gz 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37264.log.gz 2024-05-06T00:31:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36345.log 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40028.log 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31710.log.gz 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36345.log.gz 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32044.log 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40028.log.gz 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32044.log.gz 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36256.log 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gz 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38422.log 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29037.log 2024-05-06T00:31:19.771 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36256.log.gz 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38422.log.gz 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35501.log 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29037.log.gz 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32073.log 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39630.log 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35501.log.gz 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32073.log.gz 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33756.log 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28732.log 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39630.log.gz 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33756.log.gz 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28609.log 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30980.log 2024-05-06T00:31:19.772 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28732.log.gz 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28609.log.gz 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39395.log 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32144.log 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30980.log.gz 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39395.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.39395.log.gz -- 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.31216.log 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40540.log 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32144.log.gz 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31216.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31216.log.gz -5 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29432.log 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33627.log 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40540.log.gz 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29432.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-06T00:31:19.773 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29432.log.gz 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29285.log 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33627.log.gz 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33495.log 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35453.log 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29285.log.gz 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33332.log 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33495.log.gz 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35525.log 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35453.log.gz 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29311.log 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33332.log.gz 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31098.log 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35525.log.gz 2024-05-06T00:31:19.774 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28907.log 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29311.log.gz 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-client.admin.31098.log: --verbose -- /var/log/ceph/ceph-client.admin.29435.log 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31098.log.gz 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30706.log 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28907.log.gz 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35396.log 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29435.log.gz 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35330.log 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30706.log.gz 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39604.log 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35396.log.gz 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35330.log.gz 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39500.log 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30116.log 2024-05-06T00:31:19.775 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39604.log.gz 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31958.log 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39500.log.gz 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33235.log 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30116.log.gz 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31958.log.gz 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36572.log 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28502.log 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33235.log.gz 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37658.log 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36572.log.gz 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34533.log 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28502.log.gz 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37658.log.gz 2024-05-06T00:31:19.776 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40280.log 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31189.log 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34533.log.gz 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30541.log 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40280.log.gz 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29363.log 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31189.log.gz 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30541.log.gz 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38047.log 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34974.log 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29363.log.gz 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36311.log 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38047.log.gz 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34213.log 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34974.log.gz 2024-05-06T00:31:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36311.log.gz 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30083.log 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34213.log.gz 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33132.log 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31378.log 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30083.log.gz 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28635.log 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33132.log.gz 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40254.log 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31378.log.gz 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29552.log 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28635.log.gz 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33657.log 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40254.log.gz 2024-05-06T00:31:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33449.log 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29552.log.gz 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38997.log 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33657.log.gz 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33449.log.gz 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35477.log 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37095.log 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38997.log.gz 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34399.log 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35477.log.gz 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33540.log 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37095.log.gz 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30874.log 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34399.log.gz 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33540.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33540.log.gz 2024-05-06T00:31:19.779 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30611.log 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30874.log.gz 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34041.log 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29035.log 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30611.log.gz 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35293.log 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34041.log.gz 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30924.log 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29035.log.gz 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30974.log 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35293.log.gz 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35763.log 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30924.log.gz 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32019.log 2024-05-06T00:31:19.780 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30974.log.gz 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35763.log.gz 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38705.log 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32019.log.gz 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34428.log 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38317.log 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38705.log.gz 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30152.log 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34428.log.gz 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34569.log 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38317.log.gz 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24003.log 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30152.log.gz 2024-05-06T00:31:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30053.log 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34569.log.gz 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.24003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34907.log 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24003.log.gz 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38843.log 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30053.log.gz 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39656.log 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34907.log.gz 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38843.log: -- /var/log/ceph/ceph-client.admin.33031.log 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38843.log.gz 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39656.log.gz 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29651.log 2024-05-06T00:31:19.782 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33031.log.gz 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29181.log 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29651.log.gz 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29181.log.gz 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40723.log: -- /var/log/ceph/ceph-client.admin.33694.log 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31705.log 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38451.log 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33694.log.gz 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35959.log 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31705.log.gz 2024-05-06T00:31:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40514.log 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38451.log.gz 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37367.log 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35959.log.gz 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31341.log 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40514.log.gz 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40024.log 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37367.log.gz 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32148.log 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31341.log.gz 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33277.log 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40024.log.gz 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31461.log 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32148.log.gz 2024-05-06T00:31:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38132.log 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33277.log.gz 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39369.log 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31461.log.gz 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38132.log.gz 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35114.log 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38355.log 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39369.log.gz 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37704.log 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35114.log.gz 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39863.log 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38355.log.gz 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23886.log 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37704.log.gz 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32880.log 2024-05-06T00:31:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39863.log.gz 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.23886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35933.log 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23886.log.gz 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32625.log 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32880.log.gz 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30219.log 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr: 28.8% -- replaced with /var/log/ceph/ceph-client.admin.35933.log.gz 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37751.log 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32625.log.gz 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30928.log 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30219.log.gz 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37751.log.gz 2024-05-06T00:31:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39682.log 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30928.log.gz 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33793.log 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32952.log 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39682.log.gz 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29869.log 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33793.log.gz 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30771.log 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32952.log.gz 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39050.log 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29869.log: /var/log/ceph/ceph-client.admin.30771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29869.log.gz 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39447.log 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30771.log.gz 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39050.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39050.log.gz -5 2024-05-06T00:31:19.787 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40228.log 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28726.log 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39447.log.gz 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40566.log 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40228.log.gz 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34490.log 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28726.log.gz 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36801.log 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40566.log.gz 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38697.log 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34490.log.gz 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36801.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39722.log 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36801.log.gz 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36481.log 2024-05-06T00:31:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34725.log 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39722.log.gz 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36481.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36481.log.gz 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32521.log 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29964.log 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34725.log.gz 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33328.log 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32521.log.gz 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31529.log 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29964.log.gz 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29207.log 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33328.log.gz 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38812.log 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31529.log.gz 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29207.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37070.log 2024-05-06T00:31:19.789 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29207.log.gz 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31533.log 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38812.log.gz 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29025.log 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37070.log.gz 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.31533.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34626.log 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31533.log.gz 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40462.log 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29025.log.gz 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36083.log 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34626.log.gz 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40462.log.gz 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33839.log 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28729.log 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36083.log.gz 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34823.log 2024-05-06T00:31:19.790 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33839.log.gz 2024-05-06T00:31:19.791 INFO:teuthology.orchestra.run.smithi146.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-06T00:31:19.861 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28729.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28729.log.gz -5 2024-05-06T00:31:19.861 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31093.log 2024-05-06T00:31:19.861 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32692.log 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34823.log.gz 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31093.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31093.log.gz 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.32692.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37536.log 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32692.log.gz 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36545.log 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.37536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32226.log 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37536.log.gz 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28725.log 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36545.log.gz 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37309.log 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32226.log.gz 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39578.log 2024-05-06T00:31:19.862 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28725.log.gz 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29707.log 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37309.log.gz 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37004.log 0.0% 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39578.log.gz 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31806.log 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29707.log.gz 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32336.log 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37004.log.gz 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31806.log: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30512.log 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31806.log.gz 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35607.log 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32336.log.gz 2024-05-06T00:31:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40120.log 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30512.log.gz 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28867.log 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35607.log.gz 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40120.log.gz 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32431.log 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36058.log 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28867.log.gz 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39974.log 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32431.log.gz 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.36058.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34932.log 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36058.log.gz 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38558.log 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39974.log.gz 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32816.log 2024-05-06T00:31:19.864 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34932.log.gz 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31145.log 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38558.log.gz 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32816.log.gz 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32884.log 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28877.log 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31145.log.gz 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.32884.log: /var/log/ceph/ceph-client.admin.38256.log 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32884.log.gz 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28877.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.28877.log.gz /var/log/ceph/ceph-client.admin.35072.log 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35796.log 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38256.log.gz 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29036.log 2024-05-06T00:31:19.865 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35072.log.gz 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31423.log 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35796.log.gz 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29036.log: /var/log/ceph/ceph-client.admin.32854.log 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29036.log.gz 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31423.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31423.log.gz 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39720.log 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32854.log.gz 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.37794.log 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37486.log 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39720.log.gz 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32276.log 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37794.log.gz 2024-05-06T00:31:19.866 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29041.log 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37486.log: /var/log/ceph/ceph-client.admin.32276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31242.log 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37486.log.gz 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32276.log.gz/var/log/ceph/ceph-client.admin.29041.log: 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29041.log.gz 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35194.log 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31242.log: /var/log/ceph/ceph-client.admin.40488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31242.log.gz 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31295.log 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35194.log.gz 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37490.log 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39474.log 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31295.log.gz 2024-05-06T00:31:19.867 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39421.log 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37490.log.gz 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30747.log 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39474.log: /var/log/ceph/ceph-client.admin.39421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32389.log 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39421.log.gz 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39474.log.gz 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30747.log.gz 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30277.log 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29065.log 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32389.log: gzip/var/log/ceph/ceph-client.admin.30277.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40592.log 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32389.log.gz 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30277.log.gz 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29065.log.gz 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38051.log 2024-05-06T00:31:19.868 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32529.log 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40592.log: 0.0%/var/log/ceph/ceph-client.admin.38051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38948.log 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40592.log.gz 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38051.log.gz 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32529.log.gz 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33208.log 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38948.log.gz 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36802.log 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31879.log 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33208.log: /var/log/ceph/ceph-client.admin.36802.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33208.log.gz 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36802.log.gz 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36857.log 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36443.log 2024-05-06T00:31:19.869 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31879.log.gz 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36857.log.gz 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38901.log 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36443.log.gz 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37877.log 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37173.log 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38901.log: /var/log/ceph/ceph-client.admin.37877.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39224.log 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37877.log.gz 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38901.log.gz 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32306.log 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37173.log.gz 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34766.log 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39224.log: /var/log/ceph/ceph-client.admin.32306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36757.log 2024-05-06T00:31:19.870 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39224.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32306.log.gz 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34766.log.gz 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37448.log 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36757.log: /var/log/ceph/ceph-client.admin.40618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30376.log 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36757.log.gz 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37448.log.gz 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30615.log 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34017.log 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30376.log: /var/log/ceph/ceph-client.admin.30615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28687.log 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30615.log.gz 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30376.log.gz 2024-05-06T00:31:19.871 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34017.log.gz 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28727.log 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29393.log 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28687.log: /var/log/ceph/ceph-client.admin.28727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30459.log 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28687.log.gz 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28727.log.gz 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29393.log.gz 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28909.log 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36008.log 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30459.log: /var/log/ceph/ceph-client.admin.28909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37977.log 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30459.log.gz 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28909.log.gz 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36008.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.36008.log.gz /var/log/ceph/ceph-client.admin.29259.log 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-06T00:31:19.872 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28558.log 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37977.log: /var/log/ceph/ceph-client.admin.29259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35168.log 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37977.log.gz 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29259.log.gz 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28558.log.gz 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28815.log 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34828.log 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35168.log: /var/log/ceph/ceph-client.admin.28815.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30413.log 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35168.log.gz 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28815.log.gz 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34828.log.gz 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29700.log 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39887.log 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30413.log.gz 2024-05-06T00:31:19.873 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38177.log 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29700.log.gz 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39723.log 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39887.log: /var/log/ceph/ceph-client.admin.38177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30178.log 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39887.log.gz 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38177.log.gz 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39723.log.gz 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39807.log 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36919.log 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30178.log: /var/log/ceph/ceph-client.admin.39807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39807.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30178.log.gz 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36919.log.gz 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40670.log 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39727.log 2024-05-06T00:31:19.874 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39884.log: /var/log/ceph/ceph-client.admin.40670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35222.log 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40670.log.gz 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39727.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40202.log 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39727.log.gz 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35222.log.gz 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35697.log 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29337.log 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40202.log: /var/log/ceph/ceph-client.admin.35697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31051.log 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40202.log.gz 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35697.log.gz 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29337.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.29337.log.gz 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-05-06T00:31:19.875 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29437.log 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31051.log: /var/log/ceph/ceph-client.admin.39260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31051.log.gz 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38075.log 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37575.log 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29437.log.gz 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37368.log 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38075.log: /var/log/ceph/ceph-client.admin.37575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34080.log 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38075.log.gz 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37575.log.gz 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37368.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37368.log.gz --verbose 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.39187.log 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34080.log: /var/log/ceph/ceph-client.admin.39187.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.40436.log 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34080.log.gz 2024-05-06T00:31:19.876 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39187.log.gz 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40096.log 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35822.log 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40436.log.gz 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29791.log 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40096.log.gz 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32480.log 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35822.log: /var/log/ceph/ceph-client.admin.29791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35884.log 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29791.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35822.log.gz 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32480.log.gz 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33947.log 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35884.log.gz 2024-05-06T00:31:19.877 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34668.log 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33947.log.gz 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28583.log 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40697.log: /var/log/ceph/ceph-client.admin.34668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29601.log 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34668.log.gz 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28583.log.gz 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28661.log 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30005.log 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36178.log 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29601.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29601.log.gz 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30005.log: -- replaced with /var/log/ceph/ceph-client.admin.28661.log.gz 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30005.log.gz 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32596.log 2024-05-06T00:31:19.878 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29826.log 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33083.log 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32596.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36178.log.gz 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32596.log.gz 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29826.log.gz 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35553.log 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24077.log 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33083.log: /var/log/ceph/ceph-client.admin.35553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33107.log 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33083.log.gz 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35553.log.gz 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.24077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24077.log.gz 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31680.log 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35549.log 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33107.log: /var/log/ceph/ceph-client.admin.31680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33107.log.gz 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35392.log 2024-05-06T00:31:19.879 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31680.log.gz 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35032.log 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35549.log.gz 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33918.log 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35392.log: /var/log/ceph/ceph-client.admin.35032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36971.log 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35392.log.gz 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35032.log.gz 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33918.log.gz 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32948.log 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29900.log 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36971.log: gzip/var/log/ceph/ceph-client.admin.32948.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29763.log 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36971.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32948.log.gz 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29900.log.gz 2024-05-06T00:31:19.880 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29763.log.gz 2024-05-06T00:31:19.895 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-06T00:31:19.916 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21955.log 2024-05-06T00:31:19.916 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21110.log 2024-05-06T00:31:19.916 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20408.log 2024-05-06T00:31:19.916 INFO:teuthology.orchestra.run.smithi186.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.admin.21955.log.gz 2024-05-06T00:31:19.916 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23348.log 2024-05-06T00:31:19.916 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21110.log: /var/log/ceph/ceph-client.admin.20408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20751.log 2024-05-06T00:31:19.916 INFO:teuthology.orchestra.run.smithi186.stderr: 95.4% -- replaced with /var/log/ceph/ceph-client.admin.21110.log.gz 2024-05-06T00:31:19.916 INFO:teuthology.orchestra.run.smithi186.stderr: 94.4% -- replaced with /var/log/ceph/ceph-client.admin.20408.log.gz 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22157.log 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23348.log: /var/log/ceph/ceph-client.admin.20751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23278.log 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.23348.log.gz 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20751.log.gz 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23357.log 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22157.log.gz 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22830.log 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.23278.log.gz 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23357.log: 76.7% -- replaced with /var/log/ceph/ceph-client.admin.23357.log.gz 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21346.log 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22830.log: gzip 75.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.22830.log.gz -- 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr: /var/log/ceph/ceph-client.admin.21214.log 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21673.log 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21346.log: /var/log/ceph/ceph-client.admin.21214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22717.log 2024-05-06T00:31:19.917 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21346.log.gz 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.21214.log.gz 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23208.log 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr: 95.4% -- replaced with /var/log/ceph/ceph-client.admin.21673.log.gz 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22961.log 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22717.log.gz 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22401.log 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.23208.log.gz 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20934.log 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22961.log.gz 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22647.log 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.22401.log.gz 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20364.log 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.20934.log.gz 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23251.log 2024-05-06T00:31:19.918 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22647.log.gz 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20504.log 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20364.log.gz 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20584.log 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.23251.log.gz 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23147.log 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr: 21.5% -- replaced with /var/log/ceph/ceph-client.admin.20504.log.gz 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22550.log 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr: 57.0% -- replaced with /var/log/ceph/ceph-client.admin.20584.log.gz 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21251.log 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22550.log: 96.1% -- replaced with /var/log/ceph/ceph-client.admin.23147.log.gz 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23138.log 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr: 54.3% -- replaced with /var/log/ceph/ceph-client.admin.22550.log.gz 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21276.log 2024-05-06T00:31:19.919 INFO:teuthology.orchestra.run.smithi186.stderr: 52.9% -- replaced with /var/log/ceph/ceph-client.admin.21251.log.gz 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20479.log 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.23138.log.gz 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22376.log 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr: 21.5% -- replaced with /var/log/ceph/ceph-client.admin.21276.log.gz 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23032.log 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr: 95.0%/var/log/ceph/ceph-client.admin.22376.log: -- replaced with /var/log/ceph/ceph-client.admin.20479.log.gz 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22200.log 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.22376.log.gz 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20681.log 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.23032.log.gz 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23172.log 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr: 75.1% -- replaced with /var/log/ceph/ceph-client.admin.22200.log.gz 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20855.log 2024-05-06T00:31:19.920 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.20681.log.gz 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23181.log 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.23172.log.gz 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21813.log 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20855.log.gz 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20301.log 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr: 52.9% -- replaced with /var/log/ceph/ceph-client.admin.23181.log.gz 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21533.log 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.21813.log.gz 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22507.log 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20301.log.gz 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20611.log 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr: 76.3% -- replaced with /var/log/ceph/ceph-client.admin.21533.log.gz 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23321.log 2024-05-06T00:31:19.921 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22507.log.gz 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21777.log 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20611.log.gz 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22410.log 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr: 74.4% -- replaced with /var/log/ceph/ceph-client.admin.23321.log.gz 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22261.log 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr: 52.4% -- replaced with /var/log/ceph/ceph-client.admin.21777.log.gz 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22656.log 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr: 35.0% -- replaced with /var/log/ceph/ceph-client.admin.22410.log.gz 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22760.log 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22261.log.gz 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22927.log 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr: 54.1% -- replaced with /var/log/ceph/ceph-client.admin.22656.log.gz 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20310.log 2024-05-06T00:31:19.922 INFO:teuthology.orchestra.run.smithi186.stderr: 57.5% -- replaced with /var/log/ceph/ceph-client.admin.22760.log.gz 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21989.log 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.22927.log.gz 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20550.log 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr: 55.4% -- replaced with /var/log/ceph/ceph-client.admin.20310.log.gz 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20961.log 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.21989.log.gz 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21171.log 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr: 57.0% -- replaced with /var/log/ceph/ceph-client.admin.20550.log.gz 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20785.log 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.20961.log.gz 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22166.log 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21171.log.gz 2024-05-06T00:31:19.923 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21205.log 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20785.log.gz 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22297.log 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.22166.log.gz 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22620.log 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21205.log.gz 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22016.log 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.22297.log.gz 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21946.log 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 52.8% -- replaced with /var/log/ceph/ceph-client.admin.22620.log.gz 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23041.log 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22016.log.gz 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21074.log 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21946.log.gz 2024-05-06T00:31:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20724.log 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr: 52.8% -- replaced with /var/log/ceph/ceph-client.admin.23041.log.gz 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20891.log 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr: 56.9% -- replaced with /var/log/ceph/ceph-client.admin.21074.log.gz 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20925.log 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr: 55.7% -- replaced with /var/log/ceph/ceph-client.admin.20724.log.gz 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22796.log 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.20891.log.gz 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22340.log 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20925.log.gz 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22059.log 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr: 52.7% -- replaced with /var/log/ceph/ceph-client.admin.22796.log.gz 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22998.log 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr: 54.2% -- replaced with /var/log/ceph/ceph-client.admin.22340.log.gz 2024-05-06T00:31:19.925 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21462.log 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr: 72.4% -- replaced with /var/log/ceph/ceph-client.admin.22059.log.gz 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21040.log 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22998.log.gz 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21594.log 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.21462.log.gz 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.20280.log 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr: 76.1% -- replaced with /var/log/ceph/ceph-client.admin.21040.log.gz 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22516.log 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21594.log.gz 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.0.20280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20442.log 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.0.20280.log.gz 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22970.log 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr: 53.2% -- replaced with /var/log/ceph/ceph-client.admin.22516.log.gz 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22857.log 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr: 55.5% -- replaced with /var/log/ceph/ceph-client.admin.20442.log.gz 2024-05-06T00:31:19.926 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22121.log 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.22970.log.gz 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23007.log 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22857.log.gz 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20470.log 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22121.log.gz 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21567.log 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.23007.log.gz 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21312.log 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.20470.log.gz 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22087.log 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.21567.log.gz 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21383.log 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21312.log.gz 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22471.log 2024-05-06T00:31:19.927 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22087.log.gz 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20830.log 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21383.log: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21383.log.gz 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21637.log 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22471.log.gz 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21101.log 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21426.log 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr: 96.0% -- replaced with /var/log/ceph/ceph-client.admin.20830.log.gz 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr: 53.4% -- replaced with /var/log/ceph/ceph-client.admin.21637.log.gz 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22191.log 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21101.log.gz 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20575.log 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr: 79.4% -- replaced with /var/log/ceph/ceph-client.admin.21426.log.gz 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22096.log 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22191.log.gz 2024-05-06T00:31:19.928 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21876.log 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20575.log.gz 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23312.log 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.22096.log.gz 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23111.log 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21876.log.gz 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22446.log 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.23312.log.gz 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22690.log 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr: 53.9% -- replaced with /var/log/ceph/ceph-client.admin.23111.log.gz 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21487.log 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr: 53.3% -- replaced with /var/log/ceph/ceph-client.admin.22446.log.gz 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21664.log 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr: 53.2% -- replaced with /var/log/ceph/ceph-client.admin.22690.log.gz 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21734.log 2024-05-06T00:31:19.929 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21487.log.gz 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21031.log 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21664.log.gz 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20864.log 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21734.log.gz 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21558.log 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21031.log.gz 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21417.log 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr: 53.9% -- replaced with /var/log/ceph/ceph-client.admin.20864.log.gz 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22480.log 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21558.log.gz 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22891.log 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21417.log.gz 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21524.log 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr: 56.5% -- replaced with /var/log/ceph/ceph-client.admin.22480.log.gz 2024-05-06T00:31:19.930 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23102.log 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22891.log.gz 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21355.log 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21524.log.gz 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20654.log 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.23102.log.gz 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21698.log 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr: 74.6% -- replaced with /var/log/ceph/ceph-client.admin.21355.log.gz 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20620.log 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.20654.log.gz 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22437.log 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21698.log.gz 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22236.log 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22437.log: 96.2% -- replaced with /var/log/ceph/ceph-client.admin.20620.log.gz 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21628.log 2024-05-06T00:31:19.931 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22437.log.gz 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21144.log 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.22236.log.gz 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21885.log 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21628.log.gz 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23242.log 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr: 51.9% -- replaced with /var/log/ceph/ceph-client.admin.21144.log.gz 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22726.log 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21910.log 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.23242.log.gz 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr: 96.7% -- replaced with /var/log/ceph/ceph-client.admin.21885.log.gz 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22681.log 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.22726.log.gz 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21804.log 2024-05-06T00:31:19.932 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21910.log.gz 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20970.log 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.22681.log.gz 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22787.log 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21804.log.gz 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21838.log 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.20970.log.gz 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22577.log 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22787.log.gz 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21496.log 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21838.log.gz 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22050.log 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22577.log.gz 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20433.log 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.21496.log.gz 2024-05-06T00:31:19.933 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21065.log 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22050.log.gz 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21453.log 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20433.log.gz 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22936.log 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21065.log.gz 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22270.log 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21453.log.gz 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22586.log 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr: 56.0% -- replaced with /var/log/ceph/ceph-client.admin.22936.log.gz 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23217.log 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr: 54.2% -- replaced with /var/log/ceph/ceph-client.admin.22270.log.gz 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21768.log 2024-05-06T00:31:19.934 INFO:teuthology.orchestra.run.smithi186.stderr: 94.6% -- replaced with /var/log/ceph/ceph-client.admin.22586.log.gz 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22541.log 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21135.log 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr: 96.1% -- replaced with /var/log/ceph/ceph-client.admin.23217.log.gz 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21768.log.gz 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22227.log 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22541.log.gz 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21392.log 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.21135.log.gz 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22611.log 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr: 21.5% -- replaced with /var/log/ceph/ceph-client.admin.22227.log.gz 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22900.log 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr: 83.5% -- replaced with /var/log/ceph/ceph-client.admin.21392.log.gz 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20645.log 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22611.log.gz 2024-05-06T00:31:19.935 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20373.log 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr: 54.3% -- replaced with /var/log/ceph/ceph-client.admin.22900.log.gz 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20399.log 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20645.log.gz 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21919.log 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.20373.log.gz 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21242.log 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.20399.log.gz 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20995.log 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr: 53.4% -- replaced with /var/log/ceph/ceph-client.admin.21919.log.gz 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22866.log 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.21242.log.gz 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20821.log 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20995.log.gz 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22130.log 2024-05-06T00:31:19.936 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22866.log: 51.3% -- replaced with /var/log/ceph/ceph-client.admin.22866.log.gz 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20794.log 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20821.log.gz 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23077.log 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr: 35.6% -- replaced with /var/log/ceph/ceph-client.admin.22130.log.gz 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21180.log 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr: 56.9% -- replaced with /var/log/ceph/ceph-client.admin.20794.log.gz 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22025.log 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr: 95.4% -- replaced with /var/log/ceph/ceph-client.admin.23077.log.gz 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21707.log 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.21180.log.gz 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22367.log 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.22025.log.gz 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22821.log 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr: 53.6% -- replaced with /var/log/ceph/ceph-client.admin.21707.log.gz 2024-05-06T00:31:19.937 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22306.log 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22367.log.gz 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21743.log 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22821.log.gz 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23287.log 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr: 52.5% -- replaced with /var/log/ceph/ceph-client.admin.22306.log.gz 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21847.log 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.21743.log.gz 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20690.log 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.23287.log.gz 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21321.log 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr: 73.4% -- replaced with /var/log/ceph/ceph-client.admin.21847.log.gz 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20513.log 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.20690.log.gz 2024-05-06T00:31:19.938 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21285.log 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr: 94.6% -- replaced with /var/log/ceph/ceph-client.admin.21321.log.gz 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23068.log 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr: 57.3% -- replaced with /var/log/ceph/ceph-client.admin.20513.log.gz 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20900.log 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr: 56.9% -- replaced with /var/log/ceph/ceph-client.admin.21285.log.gz 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20715.log 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.23068.log.gz 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21004.log 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22331.log 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr: 95.6% -- replaced with /var/log/ceph/ceph-client.admin.20900.log.gz 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20715.log.gz 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22751.log 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr: 54.4% -- replaced with /var/log/ceph/ceph-client.admin.21004.log.gz 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21603.log 2024-05-06T00:31:19.939 INFO:teuthology.orchestra.run.smithi186.stderr: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.22331.log.gz 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.22751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21980.log 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.22751.log.gz 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20541.log 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.21603.log.gz 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20760.log 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.21980.log: 20.2% -- replaced with /var/log/ceph/ceph-client.admin.21980.log.gz 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20541.log: 20.9% -- replaced with /var/log/ceph/ceph-client.admin.20541.log.gz 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.20760.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.20760.log.gz 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m0.168s 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m0.154s 2024-05-06T00:31:19.940 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.058s 2024-05-06T00:31:20.020 INFO:teuthology.orchestra.run.smithi078.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-06T00:31:20.055 INFO:teuthology.orchestra.run.smithi078.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-06T00:31:21.596 INFO:teuthology.orchestra.run.smithi146.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-06T00:31:23.506 INFO:teuthology.orchestra.run.smithi146.stderr: 95.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-06T00:31:24.672 INFO:teuthology.orchestra.run.smithi078.stderr: 95.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-06T00:31:25.304 INFO:teuthology.orchestra.run.smithi146.stderr: 95.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-06T00:31:25.305 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-05-06T00:31:25.305 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m5.613s 2024-05-06T00:31:25.306 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m11.019s 2024-05-06T00:31:25.306 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.534s 2024-05-06T00:31:25.601 INFO:teuthology.orchestra.run.smithi078.stderr: 95.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-06T00:31:25.999 INFO:teuthology.orchestra.run.smithi078.stderr: 95.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-06T00:31:26.001 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-06T00:31:26.001 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m6.337s 2024-05-06T00:31:26.001 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m17.234s 2024-05-06T00:31:26.001 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.798s 2024-05-06T00:31:26.001 INFO:tasks.ceph:Archiving logs... 2024-05-06T00:31:26.002 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi078/log 2024-05-06T00:31:26.003 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T00:31:28.011 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi146/log 2024-05-06T00:31:28.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T00:31:29.285 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi186/log 2024-05-06T00:31:29.286 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T00:31:29.429 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-06T00:31:29.443 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-06T00:31:29.444 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-06T00:31:29.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-06T00:31:29.449 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-06T00:31:29.540 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-06T00:31:29.549 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-06T00:31:29.549 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T00:31:29.552 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T00:31:29.554 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T00:31:29.563 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2024-05-06T00:31:29.563 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2024-05-06T00:31:29.563 INFO:teuthology.orchestra.run.smithi078.stdout:+hv01.front.sepi 67.205.162.81 3 u 2 64 377 0.046 -2.957 1.834 2024-05-06T00:31:29.563 INFO:teuthology.orchestra.run.smithi078.stdout:*hv02.front.sepi 63.231.80.2 3 u 5 64 377 0.093 +0.983 1.333 2024-05-06T00:31:29.563 INFO:teuthology.orchestra.run.smithi078.stdout:+hv03.front.sepi 74.6.168.72 3 u 1 64 377 0.086 +0.311 1.325 2024-05-06T00:31:29.563 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:31:29.564 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2024-05-06T00:31:29.564 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2024-05-06T00:31:29.564 INFO:teuthology.orchestra.run.smithi146.stdout:+hv01.front.sepi 67.205.162.81 3 u 5 64 377 0.077 -1.891 0.492 2024-05-06T00:31:29.564 INFO:teuthology.orchestra.run.smithi146.stdout:+hv02.front.sepi 63.231.80.2 3 u 2 64 377 0.049 +0.455 0.698 2024-05-06T00:31:29.565 INFO:teuthology.orchestra.run.smithi146.stdout:*hv03.front.sepi 74.6.168.72 3 u 6 64 377 0.096 -0.855 0.641 2024-05-06T00:31:29.565 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:31:29.567 INFO:teuthology.orchestra.run.smithi186.stdout: remote refid st t when poll reach delay offset jitter 2024-05-06T00:31:29.567 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================== 2024-05-06T00:31:29.568 INFO:teuthology.orchestra.run.smithi186.stdout:+hv01.front.sepi 67.205.162.81 3 u 10 64 377 0.081 -1.424 0.870 2024-05-06T00:31:29.568 INFO:teuthology.orchestra.run.smithi186.stdout:*hv02.front.sepi 63.231.80.2 3 u 6 64 377 0.088 +0.234 0.656 2024-05-06T00:31:29.568 INFO:teuthology.orchestra.run.smithi186.stdout:+hv03.front.sepi 74.6.168.72 3 u 6 64 377 0.090 -1.738 1.420 2024-05-06T00:31:29.568 INFO:teuthology.orchestra.run.smithi186.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-06T00:31:29.569 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-06T00:31:29.578 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-06T00:31:29.578 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-06T00:31:29.586 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-06T00:31:29.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-06T00:31:29.650 INFO:teuthology.task.internal:Duration was 896.904090 seconds 2024-05-06T00:31:29.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-06T00:31:29.659 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-06T00:31:29.659 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T00:31:29.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T00:31:29.664 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T00:31:29.700 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-06T00:31:29.700 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-05-06T00:31:29.701 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T00:31:29.753 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-05-06T00:31:29.753 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T00:31:29.763 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-05-06T00:31:29.763 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T00:31:29.773 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-06T00:31:29.773 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T00:31:29.797 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T00:31:29.809 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T00:31:29.845 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-06T00:31:29.845 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T00:31:29.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T00:31:29.877 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T00:31:29.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-06T00:31:29.960 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-06T00:31:29.960 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T00:31:29.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T00:31:29.984 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T00:31:29.996 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-06T00:31:30.006 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T00:31:30.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T00:31:30.025 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-05-06T00:31:30.030 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T00:31:30.038 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-05-06T00:31:30.048 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-05-06T00:31:30.121 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T00:31:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:31:30.127 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T00:31:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:31:30.144 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T00:31:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T00:31:30.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-06T00:31:30.181 INFO:teuthology.task.internal:Transferring archived files... 2024-05-06T00:31:30.182 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi078 2024-05-06T00:31:30.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T00:31:30.205 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi146 2024-05-06T00:31:30.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T00:31:30.226 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-05_18:40:04-rbd-wip-dis-testing-distro-default-smithi/7692201/remote/smithi186 2024-05-06T00:31:30.226 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T00:31:30.251 INFO:teuthology.task.internal:Removing archive directory... 2024-05-06T00:31:30.252 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T00:31:30.255 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T00:31:30.270 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T00:31:30.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-06T00:31:30.302 INFO:teuthology.task.internal:Not uploading archives. 2024-05-06T00:31:30.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-06T00:31:30.312 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-06T00:31:30.312 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T00:31:30.315 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T00:31:30.317 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T00:31:30.318 INFO:teuthology.orchestra.run.smithi078.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 00:31 /home/ubuntu/cephtest 2024-05-06T00:31:30.321 INFO:teuthology.orchestra.run.smithi146.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 00:31 /home/ubuntu/cephtest 2024-05-06T00:31:30.338 INFO:teuthology.orchestra.run.smithi186.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 00:31 /home/ubuntu/cephtest 2024-05-06T00:31:30.339 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-06T00:31:30.350 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-06T00:31:30.375 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nbd} duration: 896.9040896892548 flavor: default owner: scheduled_dis@teuthology success: true 2024-05-06T00:31:30.375 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T00:31:30.457 INFO:teuthology.run:pass